2024-03-29T10:46:49.872 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:46:49.872 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --verbose -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/orig.config.yaml 2024-03-29T10:46:49.906 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:46:49.995 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622667' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFzZxd2kvuz30bPI7xAlhJL5tJ7HLRwgTx6Tm6UbCbL2tMstDFr0/oYOlXARa7OJEEd80SMqSR3NCkqnlii78Q= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNHbCoWq3xgvmIDZlynuq9vF0SpeCEjLpTRbMDfDt5Q/x/9Wy9L9q5yC27U+4CL6J7Pr+MJ8ognYTedVfMR8So= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:46:49.996 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:46:49.996 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:46:49.996 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:46:49.998 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:46:50.024 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:46:50.025 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:46:50.025 INFO:teuthology.packaging:ref: None 2024-03-29T10:46:50.025 INFO:teuthology.packaging:tag: None 2024-03-29T10:46:50.025 INFO:teuthology.packaging:branch: main 2024-03-29T10:46:50.025 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:46:50.025 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:46:50.410 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:46:50.412 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:46:50.419 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:46:50.419 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:46:50.456 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:46:50.471 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:46:50.490 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:46:50.513 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:41:44.155932', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFzZxd2kvuz30bPI7xAlhJL5tJ7HLRwgTx6Tm6UbCbL2tMstDFr0/oYOlXARa7OJEEd80SMqSR3NCkqnlii78Q='} 2024-03-29T10:46:50.534 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:41:44.157686', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNHbCoWq3xgvmIDZlynuq9vF0SpeCEjLpTRbMDfDt5Q/x/9Wy9L9q5yC27U+4CL6J7Pr+MJ8ognYTedVfMR8So='} 2024-03-29T10:46:50.534 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:46:50.541 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:46:50.541 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:46:50.542 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:46:50.660 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbc14dc60d0>, signals=[15]) 2024-03-29T10:46:50.660 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:46:50.667 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:46:50.667 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T10:46:50.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:46:50.762 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T10:46:50.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:46:50.870 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:46:50.877 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-03-29T10:46:50.894 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-03-29T10:46:50.894 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-03-29T10:46:50.950 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-03-29T10:46:50.950 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:46:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:46:50.952 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-03-29T10:46:50.972 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-03-29T10:46:50.989 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-03-29T10:46:50.989 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-03-29T10:46:51.045 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-03-29T10:46:51.045 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-03-29T10:46:51.045 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-03-29T10:46:51.045 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:46:51.045 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-03-29T10:46:51.046 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:46:51.046 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:46:51.046 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-03-29T10:46:51.046 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:46:51.046 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:46:51.046 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:46:51.046 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:46:51.046 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:46:51.046 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:46:51.046 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-03-29T10:46:51.071 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:46:51.080 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:46:51.107 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:46:51.107 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:46:51.110 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:46:51.125 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:46:51.140 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:46:51.140 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:46:51.166 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:46:51.196 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:46:51.251 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:46:51.266 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:46:51.266 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:46:51.266 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:46:51.267 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:46:51.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:46:51.282 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-29T10:46:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:46:51.283 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-03-29T10:46:51.338 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:46:51.338 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-361.el9.x86_64 2024-03-29T10:46:51.339 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-03-29T10:46:51.340 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:46:51.340 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-361.el9.x86_64 2024-03-29T10:46:51.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-03-29T10:46:53.418 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T10:46:53.511 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T10:46:54.802 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-29T10:46:55.144 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-29T10:46:56.137 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 8.0 MB/s | 8.0 MB 00:01 2024-03-29T10:46:57.460 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-03-29T10:46:58.402 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-29T10:46:59.302 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-29T10:47:02.361 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-29T10:47:03.932 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-29T10:47:04.644 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.1 MB 00:01 2024-03-29T10:47:05.765 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T10:47:06.053 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-29T10:47:07.279 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-29T10:47:10.730 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-29T10:47:12.488 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 5.1 kB/s | 1.7 kB 00:00 2024-03-29T10:47:13.003 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:47:13.003 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:47:13.101 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:47:13.111 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:47:13.111 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:47:13.111 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:47:13.111 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:47:13.111 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:47:13.111 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:47:13.111 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:47:13.111 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:47:13.111 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:47:13.112 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:47:13.112 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:47:13.112 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:47:13.112 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-03-29T10:47:13.112 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:47:13.113 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 76 M 2024-03-29T10:47:13.113 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 123 M 2024-03-29T10:47:13.113 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:47:13.256 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:47:13.856 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-29T10:47:14.390 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 26 MB/s | 34 MB 00:01 2024-03-29T10:47:14.719 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:47:14.719 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:47:14.765 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-29T10:47:14.768 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:14.768 INFO:teuthology.orchestra.run.smithi129.stdout:Total 46 MB/s | 76 MB 00:01 2024-03-29T10:47:14.768 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:47:14.812 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:47:14.819 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:47:14.819 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:47:14.820 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-03-29T10:47:14.821 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:47:14.821 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 76 M 2024-03-29T10:47:14.821 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 123 M 2024-03-29T10:47:14.822 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:47:14.972 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T10:47:15.142 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:47:15.142 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:47:15.555 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T10:47:15.975 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:47:15.976 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:47:16.222 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T10:47:16.556 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T10:47:16.559 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:16.559 INFO:teuthology.orchestra.run.smithi107.stdout:Total 44 MB/s | 76 MB 00:01 2024-03-29T10:47:16.559 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:47:16.948 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:47:16.948 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:47:17.780 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:47:17.780 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:47:18.277 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:47:18.864 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:47:18.940 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:47:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:47:20.677 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:47:20.898 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:47:21.694 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:47:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:47:23.502 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:47:23.641 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:47:26.588 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:47:26.630 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:47:28.332 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:47:28.373 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:47:31.346 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:47:33.084 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:48:00.722 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:48:00.722 INFO:teuthology.orchestra.run.smithi129.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:48:00.722 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:48:00.765 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:48:01.123 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:48:01.123 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:48:01.123 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:48:01.123 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:48:01.558 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:48:01.558 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:48:01.559 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:48:01.559 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:48:01.559 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:48:01.559 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:48:01.559 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:48:01.559 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:48:01.559 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:48:01.894 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:48:01.997 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:48:01.998 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:48:01.998 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:48:01.998 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:48:01.998 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:48:01.998 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:48:01.998 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:48:01.998 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:48:02.080 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:48:02.080 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:48:02.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-03-29T10:48:02.118 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-03-29T10:48:02.118 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-03-29T10:48:02.118 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-03-29T10:48:02.118 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 68.el9 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8264254 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-03-29T10:48:02.119 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:48:02.120 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-03-29T10:48:02.120 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-03-29T10:48:02.120 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:48:02.120 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:48:02.120 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:48:02.120 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:48:02.120 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-03-29T10:48:02.120 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:48:02.120 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:48:02.121 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:48:02.121 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:48:02.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:48:02.551 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:48:02.551 INFO:teuthology.orchestra.run.smithi107.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:48:02.551 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:48:02.595 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:48:02.768 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-03-29T10:48:03.009 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:48:03.009 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:48:03.009 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:48:03.009 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:48:03.432 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:48:03.432 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:48:03.432 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:48:03.432 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:48:03.432 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:48:03.432 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:48:03.432 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:48:03.432 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:48:03.432 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:48:03.802 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:48:03.919 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:48:03.920 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:48:03.920 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:48:03.920 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:48:03.920 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:48:03.920 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:48:03.920 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:48:03.920 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:48:04.000 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:48:04.001 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:48:04.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 68.el9 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8264254 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-03-29T10:48:04.034 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:48:04.035 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:48:04.036 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:48:04.036 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:48:04.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:48:04.072 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:48:04.101 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-03-29T10:48:04.103 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-03-29T10:48:04.119 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.CIb27RpyPb 2024-03-29T10:48:04.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CIb27RpyPb 2024-03-29T10:48:04.188 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.CIb27RpyPb 2024-03-29T10:48:04.330 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.CIb27RpyPb is 6KB 2024-03-29T10:48:04.378 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.CIb27RpyPb 2024-03-29T10:48:04.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:48:04.457 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:48:04.457 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:48:04.457 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:48:04.457 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:48:04.458 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:48:04.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-03-29T10:48:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-03-29T10:48:06.077 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:48:06.097 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-03-29T10:48:06.099 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-03-29T10:48:06.114 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.oKRbza5KuB 2024-03-29T10:48:06.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oKRbza5KuB 2024-03-29T10:48:06.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.oKRbza5KuB 2024-03-29T10:48:06.325 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.oKRbza5KuB is 6KB 2024-03-29T10:48:06.374 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.oKRbza5KuB 2024-03-29T10:48:06.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:48:06.458 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:48:06.459 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:48:06.459 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:48:06.459 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:48:06.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:48:06.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-03-29T10:48:34.626 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:48:34.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-29T10:48:34.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:48:36.618 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:48:36.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-29T10:48:36.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:48:53.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-29T10:48:55.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-03-29T10:49:02.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-29T10:49:02.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:49:04.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-29T10:49:04.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:49:05.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-29T10:49:07.138 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-03-29T10:49:17.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-29T10:49:17.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:49:19.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-29T10:49:19.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:49:37.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-03-29T10:49:48.983 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-29T10:49:49.403 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-29T10:49:49.403 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:49:49.404 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-29T10:49:49.418 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:49:49.418 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:49:49.418 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:49:49.418 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:49:50.419 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:49:50.419 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-03-29T10:49:50.531 DEBUG:teuthology.parallel:result is None 2024-03-29T10:49:52.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-29T10:49:52.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:49:52.987 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-03-29T10:49:53.445 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-29T10:49:53.445 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:49:53.445 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-03-29T10:49:53.459 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:49:53.459 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:49:53.459 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:49:53.459 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:49:54.461 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:49:54.461 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:49:54.532 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-03-29T10:49:54.595 DEBUG:teuthology.parallel:result is None 2024-03-29T10:49:54.595 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:49:54.602 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:49:54.602 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:49:54.604 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:49:54.623 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:49:54.639 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:49:54.681 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:49:54.682 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:49:54.684 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:49:54.713 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:49:54.731 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:49:54.731 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:49:54.746 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:49:54.781 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:49:54.781 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:49:54.794 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:49:54.795 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:49:54.797 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:49:54.804 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:49:54.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:49:54.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:49:54.870 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:49:54.878 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:49:54.878 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:49:54.909 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:49:54.926 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:49:55.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:49:55.074 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:49:55.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:49:55.176 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:49:55.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:49:55.263 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:49:55.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:49:55.366 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:49:55.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:49:55.493 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:49:55.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:49:55.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-03-29T10:49:55.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-03-29T10:49:55.637 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:49:55.649 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:49:56.158 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:49:56.166 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:49:56.166 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:49:56.210 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:49:56.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-03-29T10:49:56.312 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-03-29T10:49:56.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-03-29T10:49:56.370 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-03-29T10:49:56.372 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:49:56.372 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-03-29T10:49:56.531 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-03-29T10:49:56.531 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-03-29T10:49:56.556 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-03-29T10:49:56.557 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi129.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:49:56.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:49:56.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:49:56.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:49:56.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:49:56.625 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:49:56.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:49:56.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:49:56.750 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:49:56.760 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T10:49:56.765 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T10:49:56.900 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:49:56.920 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:49:56.922 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:54:14.625 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-03-29T10:54:14.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-29T10:54:14.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:54:14.721 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-03-29T10:54:14.802 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-29T10:54:14.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-29T10:54:14.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:54:14.895 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-29T10:54:14.973 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-29T10:54:14.973 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:54:14.984 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:54:14.985 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:54:14.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:54:14.987 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:54:14.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:54:15.016 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:54:15.033 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:54:15.044 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:54:15.061 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:54:15.062 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-03-29T10:54:15.076 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-03-29T10:54:15.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-03-29T10:54:15.092 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:54:15.105 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-03-29T10:54:15.108 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:54:15.121 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:54:15.138 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:54:15.162 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:54:15.166 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:54:15.167 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-29T10:54:15.167 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:54:15.167 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:54:15.167 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:54:15.167 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:54:15.192 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:54:15.196 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:54:15.196 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-29T10:54:15.196 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:54:15.196 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:54:15.196 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:54:15.196 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:54:15.197 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:54:15.205 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:54:15.205 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:54:15.205 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:54:15.205 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:54:15.211 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:54:15.211 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:54:15.211 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:54:15.212 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:54:15.214 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:54:15.214 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:54:15.430 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:54:15.431 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:54:15.435 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:54:15.436 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:54:15.600 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:54:15.600 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:54:15.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:54:15.606 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:54:15.606 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:54:15.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:54:15.630 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:54:15.630 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:54:15.635 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:54:15.636 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:54:15.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:54:15.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:54:15.780 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:54:15.784 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:54:15.846 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-03-29T10:54:15.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-03-29T10:54:15.851 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-03-29T10:54:15.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-03-29T10:54:16.097 INFO:teuthology.orchestra.run.smithi129.stdout:74 files removed 2024-03-29T10:54:16.097 INFO:teuthology.orchestra.run.smithi107.stdout:74 files removed 2024-03-29T10:54:16.124 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw 2024-03-29T10:54:16.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2024-03-29T10:54:16.846 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 196 kB/s | 82 kB 00:00 2024-03-29T10:54:16.898 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 174 kB/s | 82 kB 00:00 2024-03-29T10:54:17.221 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-29T10:54:17.323 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-29T10:54:17.630 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-29T10:54:17.715 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-03-29T10:54:18.249 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T10:54:18.351 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T10:54:19.733 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-29T10:54:19.734 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-29T10:54:20.657 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-29T10:54:22.367 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.0 MB 00:02 2024-03-29T10:54:22.483 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-29T10:54:26.628 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.1 MB 00:00 2024-03-29T10:54:27.334 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 19 MB 00:03 2024-03-29T10:54:28.435 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-29T10:54:30.389 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T10:54:31.403 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T10:54:32.820 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-29T10:54:34.338 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-29T10:54:35.245 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-29T10:54:37.846 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:54:37.850 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:54:37.850 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:54:37.850 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:54:37.850 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:54:37.850 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:54:37.850 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:54:37.851 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:54:37.852 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout:Install 30 Packages 2024-03-29T10:54:37.853 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 2 Packages 2024-03-29T10:54:37.854 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:54:37.854 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 58 M 2024-03-29T10:54:37.854 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:54:39.271 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-29T10:54:39.305 INFO:teuthology.orchestra.run.smithi129.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 14 MB/s | 20 MB 00:01 2024-03-29T10:54:39.347 INFO:teuthology.orchestra.run.smithi129.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 596 kB/s | 25 kB 00:00 2024-03-29T10:54:39.448 INFO:teuthology.orchestra.run.smithi129.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.2 MB/s | 742 kB 00:00 2024-03-29T10:54:39.515 INFO:teuthology.orchestra.run.smithi129.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T10:54:39.649 INFO:teuthology.orchestra.run.smithi129.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 5.7 MB/s | 9.7 MB 00:01 2024-03-29T10:54:39.699 INFO:teuthology.orchestra.run.smithi129.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 907 kB/s | 45 kB 00:00 2024-03-29T10:54:39.791 INFO:teuthology.orchestra.run.smithi129.stdout:(7/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 19 MB/s | 5.1 MB 00:00 2024-03-29T10:54:39.816 INFO:teuthology.orchestra.run.smithi129.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.1 MB/s | 131 kB 00:00 2024-03-29T10:54:39.841 INFO:teuthology.orchestra.run.smithi129.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.1 MB/s | 162 kB 00:00 2024-03-29T10:54:39.866 INFO:teuthology.orchestra.run.smithi129.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 6.3 MB/s | 320 kB 00:00 2024-03-29T10:54:39.892 INFO:teuthology.orchestra.run.smithi129.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.8 MB/s | 298 kB 00:00 2024-03-29T10:54:39.917 INFO:teuthology.orchestra.run.smithi129.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:54:40.076 INFO:teuthology.orchestra.run.smithi129.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 576 kB/s | 106 kB 00:00 2024-03-29T10:54:40.118 INFO:teuthology.orchestra.run.smithi129.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 795 kB/s | 159 kB 00:00 2024-03-29T10:54:40.143 INFO:teuthology.orchestra.run.smithi129.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 663 kB/s | 44 kB 00:00 2024-03-29T10:54:40.210 INFO:teuthology.orchestra.run.smithi129.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-29T10:54:40.235 INFO:teuthology.orchestra.run.smithi129.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-29T10:54:40.252 INFO:teuthology.orchestra.run.smithi129.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 989 kB/s | 41 kB 00:00 2024-03-29T10:54:40.278 INFO:teuthology.orchestra.run.smithi129.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T10:54:40.295 INFO:teuthology.orchestra.run.smithi129.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 840 kB/s | 35 kB 00:00 2024-03-29T10:54:40.311 INFO:teuthology.orchestra.run.smithi129.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.3 MB/s | 46 kB 00:00 2024-03-29T10:54:40.478 INFO:teuthology.orchestra.run.smithi129.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-29T10:54:40.554 INFO:teuthology.orchestra.run.smithi129.stdout:(23/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 2.0 MB/s | 5.3 MB 00:02 2024-03-29T10:54:40.570 INFO:teuthology.orchestra.run.smithi129.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 282 kB/s | 26 kB 00:00 2024-03-29T10:54:40.612 INFO:teuthology.orchestra.run.smithi129.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-29T10:54:40.646 INFO:teuthology.orchestra.run.smithi129.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 526 kB/s | 49 kB 00:00 2024-03-29T10:54:40.755 INFO:teuthology.orchestra.run.smithi129.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-29T10:54:40.805 INFO:teuthology.orchestra.run.smithi129.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-29T10:54:40.839 INFO:teuthology.orchestra.run.smithi129.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 996 kB/s | 192 kB 00:00 2024-03-29T10:54:40.881 INFO:teuthology.orchestra.run.smithi129.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-29T10:54:41.081 INFO:teuthology.orchestra.run.smithi129.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 13 MB/s | 3.1 MB 00:00 2024-03-29T10:54:41.739 INFO:teuthology.orchestra.run.smithi129.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.6 MB/s | 3.4 MB 00:00 2024-03-29T10:54:41.743 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:54:41.744 INFO:teuthology.orchestra.run.smithi129.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T10:54:41.744 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:54:41.782 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:54:41.782 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:54:41.894 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:54:41.912 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:54:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:54:41.912 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:54:41.912 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:54:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:54:41.912 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-03-29T10:54:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:54:41.912 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:54:41.913 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse 2024-03-29T10:54:41.914 INFO:teuthology.orchestra.run.smithi107.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:54:41.915 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:54:41.916 INFO:teuthology.orchestra.run.smithi107.stdout:Install 30 Packages 2024-03-29T10:54:41.916 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2024-03-29T10:54:41.916 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:54:41.916 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 58 M 2024-03-29T10:54:41.916 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:54:42.249 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:54:42.249 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:54:42.880 INFO:teuthology.orchestra.run.smithi107.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 6.6 MB/s | 5.3 MB 00:00 2024-03-29T10:54:42.930 INFO:teuthology.orchestra.run.smithi107.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 503 kB/s | 25 kB 00:00 2024-03-29T10:54:43.047 INFO:teuthology.orchestra.run.smithi107.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.2 MB/s | 742 kB 00:00 2024-03-29T10:54:43.156 INFO:teuthology.orchestra.run.smithi107.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.5 MB/s | 498 kB 00:00 2024-03-29T10:54:43.722 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:54:43.764 INFO:teuthology.orchestra.run.smithi107.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.4 MB/s | 5.1 MB 00:00 2024-03-29T10:54:43.801 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:54:43.836 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:54:43.839 INFO:teuthology.orchestra.run.smithi107.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 603 kB/s | 45 kB 00:00 2024-03-29T10:54:43.860 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:54:43.884 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:54:43.901 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:54:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:(7/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 10 MB/s | 20 MB 00:01 2024-03-29T10:54:44.065 INFO:teuthology.orchestra.run.smithi107.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 584 kB/s | 131 kB 00:00 2024-03-29T10:54:44.090 INFO:teuthology.orchestra.run.smithi107.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-29T10:54:44.093 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:54:44.115 INFO:teuthology.orchestra.run.smithi107.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 6.2 MB/s | 320 kB 00:00 2024-03-29T10:54:44.141 INFO:teuthology.orchestra.run.smithi107.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.8 MB/s | 298 kB 00:00 2024-03-29T10:54:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:54:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:54:44.275 INFO:teuthology.orchestra.run.smithi107.stdout:(13/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 4.4 MB/s | 9.7 MB 00:02 2024-03-29T10:54:44.408 INFO:teuthology.orchestra.run.smithi107.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 396 kB/s | 106 kB 00:00 2024-03-29T10:54:44.434 INFO:teuthology.orchestra.run.smithi107.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 407 kB/s | 44 kB 00:00 2024-03-29T10:54:44.459 INFO:teuthology.orchestra.run.smithi107.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 544 kB/s | 159 kB 00:00 2024-03-29T10:54:44.478 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:54:44.501 INFO:teuthology.orchestra.run.smithi107.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-29T10:54:44.526 INFO:teuthology.orchestra.run.smithi107.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-29T10:54:44.537 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:54:44.551 INFO:teuthology.orchestra.run.smithi107.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 449 kB/s | 41 kB 00:00 2024-03-29T10:54:44.576 INFO:teuthology.orchestra.run.smithi107.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-03-29T10:54:44.602 INFO:teuthology.orchestra.run.smithi107.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-29T10:54:44.627 INFO:teuthology.orchestra.run.smithi107.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 618 kB/s | 46 kB 00:00 2024-03-29T10:54:44.629 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:54:44.721 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:54:44.747 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:54:44.793 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:54:44.822 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:54:44.866 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:54:44.885 INFO:teuthology.orchestra.run.smithi107.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 101 kB/s | 26 kB 00:00 2024-03-29T10:54:44.952 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:54:44.974 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:54:45.038 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:54:45.069 INFO:teuthology.orchestra.run.smithi107.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-03-29T10:54:45.073 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:54:45.185 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:54:45.203 INFO:teuthology.orchestra.run.smithi107.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 494 kB/s | 309 kB 00:00 2024-03-29T10:54:45.228 INFO:teuthology.orchestra.run.smithi107.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 425 kB/s | 67 kB 00:00 2024-03-29T10:54:45.233 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:54:45.273 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:54:45.305 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:54:45.334 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:54:45.367 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:54:45.445 INFO:teuthology.orchestra.run.smithi107.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 885 kB/s | 192 kB 00:00 2024-03-29T10:54:45.478 INFO:teuthology.orchestra.run.smithi107.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-03-29T10:54:45.604 INFO:teuthology.orchestra.run.smithi107.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-03-29T10:54:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:54:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:54:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:54:45.787 INFO:teuthology.orchestra.run.smithi107.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 11 MB/s | 3.4 MB 00:00 2024-03-29T10:54:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:54:45.838 INFO:teuthology.orchestra.run.smithi107.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-03-29T10:54:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:54:45.954 INFO:teuthology.orchestra.run.smithi107.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 8.9 MB/s | 3.1 MB 00:00 2024-03-29T10:54:45.959 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:54:45.959 INFO:teuthology.orchestra.run.smithi107.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-29T10:54:45.959 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:54:45.994 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:54:45.994 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:54:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:54:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:54:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:54:46.459 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:54:46.460 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:54:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:54:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:54:47.930 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:54:48.008 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:54:48.052 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:54:48.084 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:54:48.115 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:54:48.132 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:54:48.299 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:54:48.363 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:54:48.624 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:54:48.657 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:54:48.701 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:54:48.761 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:54:48.836 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:54:48.926 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:54:48.954 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:54:49.008 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:54:49.037 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:54:49.082 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:54:49.150 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:54:49.173 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:54:49.218 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:54:49.231 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:54:49.251 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:54:49.267 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:54:49.343 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:54:49.343 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:54:49.344 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:54:49.344 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:54:49.395 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:54:49.399 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:54:49.439 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:54:49.480 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:54:49.512 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:54:49.541 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:54:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:54:49.903 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:54:49.937 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:54:50.026 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:54:50.058 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:54:50.112 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:54:50.563 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:54:50.598 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:54:50.660 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:54:50.729 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:54:50.759 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:54:52.896 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:54:52.929 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:54:53.482 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:54:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:54:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:54:53.616 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:54:53.616 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:54:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:54:53.659 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:54:59.083 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:54:59.083 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-03-29T10:54:59.083 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-03-29T10:54:59.083 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-03-29T10:54:59.083 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-03-29T10:54:59.083 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-03-29T10:54:59.083 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-03-29T10:54:59.083 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-03-29T10:54:59.083 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:54:59.980 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:55:00.005 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:55:00.005 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:00.005 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:55:00.005 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:55:00.005 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:55:00.005 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:00.007 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:55:00.083 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:55:00.083 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:55:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:55:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:55:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:55:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:55:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:55:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:55:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:55:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:55:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:55:02.249 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.250 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:55:02.251 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:55:02.252 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:55:02.253 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:55:02.253 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:55:02.253 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:02.253 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:02.590 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-test 2024-03-29T10:55:03.145 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:03.236 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:55:03.236 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-03-29T10:55:03.237 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-03-29T10:55:03.237 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-03-29T10:55:03.237 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-03-29T10:55:03.237 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-03-29T10:55:03.237 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-03-29T10:55:03.237 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-03-29T10:55:03.237 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:03.303 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:55:03.304 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:55:03.305 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:55:03.305 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:03.305 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:55:03.305 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:55:03.305 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2024-03-29T10:55:03.305 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:03.305 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 48 M 2024-03-29T10:55:03.306 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 204 M 2024-03-29T10:55:03.306 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:55:03.595 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:55:03.637 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-29T10:55:03.662 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-29T10:55:03.796 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 441 kB/s | 163 kB 00:00 2024-03-29T10:55:03.879 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 862 kB/s | 187 kB 00:00 2024-03-29T10:55:04.139 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:55:04.163 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:55:04.163 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:04.163 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:55:04.163 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:55:04.163 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:55:04.164 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:04.165 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:55:04.258 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:55:04.258 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:55:05.402 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:55:05.402 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:55:05.402 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:55:05.402 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:55:05.402 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:55:05.403 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:55:05.404 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:55:05.405 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:55:05.405 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:55:05.405 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:55:05.405 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:55:06.472 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 16 MB/s | 47 MB 00:03 2024-03-29T10:55:06.475 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:06.475 INFO:teuthology.orchestra.run.smithi129.stdout:Total 15 MB/s | 48 MB 00:03 2024-03-29T10:55:06.490 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:55:06.511 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:55:06.512 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:55:06.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:55:06.532 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:06.532 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-03-29T10:55:06.532 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.532 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.532 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:55:06.532 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:55:06.532 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:55:06.533 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:55:06.534 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:55:06.535 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:55:06.535 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:55:06.535 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:06.535 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:55:06.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2024-03-29T10:55:06.887 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:55:06.888 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:55:07.362 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:55:07.472 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:55:07.520 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:55:07.521 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:55:07.522 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:07.522 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:55:07.522 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:55:07.522 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-03-29T10:55:07.522 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:07.523 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 48 M 2024-03-29T10:55:07.523 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 204 M 2024-03-29T10:55:07.523 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:55:07.537 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:55:07.582 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:55:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:55:07.671 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:55:07.690 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:55:07.962 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:55:08.012 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:55:08.045 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:55:08.079 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 593 kB/s | 163 kB 00:00 2024-03-29T10:55:08.371 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 575 kB/s | 187 kB 00:00 2024-03-29T10:55:10.381 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-29T10:55:10.382 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:10.383 INFO:teuthology.orchestra.run.smithi107.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-29T10:55:10.397 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:55:10.418 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:55:10.418 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:55:10.795 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:55:10.795 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:55:11.368 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:55:11.446 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:55:11.487 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:55:11.528 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:55:11.576 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:55:11.607 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:55:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:55:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:55:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:55:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:55:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:55:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:55:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:55:14.634 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:15.007 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:15.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph 2024-03-29T10:55:15.876 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:16.029 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:16.031 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================= 2024-03-29T10:55:16.031 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:55:16.031 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================= 2024-03-29T10:55:16.031 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:55:16.032 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:55:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================= 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout:Install 36 Packages 2024-03-29T10:55:16.034 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:16.035 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 30 M 2024-03-29T10:55:16.035 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 107 M 2024-03-29T10:55:16.035 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:55:16.105 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:55:16.181 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:55:16.783 INFO:teuthology.orchestra.run.smithi129.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-03-29T10:55:16.975 INFO:teuthology.orchestra.run.smithi129.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-03-29T10:55:17.892 INFO:teuthology.orchestra.run.smithi129.stdout:(3/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 17 MB 00:00 2024-03-29T10:55:17.925 INFO:teuthology.orchestra.run.smithi129.stdout:(4/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-03-29T10:55:17.993 INFO:teuthology.orchestra.run.smithi129.stdout:(5/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 3.9 MB/s | 4.7 MB 00:01 2024-03-29T10:55:18.043 INFO:teuthology.orchestra.run.smithi129.stdout:(6/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.6 MB/s | 247 kB 00:00 2024-03-29T10:55:18.134 INFO:teuthology.orchestra.run.smithi129.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 894 kB/s | 187 kB 00:00 2024-03-29T10:55:18.161 INFO:teuthology.orchestra.run.smithi129.stdout:(8/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2024-03-29T10:55:18.176 INFO:teuthology.orchestra.run.smithi129.stdout:(9/36): python3-markupsafe-1.1.1-12.el9.x86_64. 293 kB/s | 32 kB 00:00 2024-03-29T10:55:18.194 INFO:teuthology.orchestra.run.smithi129.stdout:(10/36): python3-pytz-2021.1-5.el9.noarch.rpm 808 kB/s | 47 kB 00:00 2024-03-29T10:55:18.210 INFO:teuthology.orchestra.run.smithi129.stdout:(11/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:55:18.244 INFO:teuthology.orchestra.run.smithi129.stdout:(12/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-29T10:55:18.319 INFO:teuthology.orchestra.run.smithi129.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 9.1 MB/s | 1.1 MB 00:00 2024-03-29T10:55:18.344 INFO:teuthology.orchestra.run.smithi129.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 771 kB/s | 103 kB 00:00 2024-03-29T10:55:18.369 INFO:teuthology.orchestra.run.smithi129.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 993 kB/s | 124 kB 00:00 2024-03-29T10:55:18.395 INFO:teuthology.orchestra.run.smithi129.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-29T10:55:18.520 INFO:teuthology.orchestra.run.smithi129.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 149 kB/s | 22 kB 00:00 2024-03-29T10:55:18.638 INFO:teuthology.orchestra.run.smithi129.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 468 kB/s | 125 kB 00:00 2024-03-29T10:55:18.662 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:55:18.662 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:55:18.662 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:55:18.662 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:55:18.662 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:55:18.662 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:55:18.779 INFO:teuthology.orchestra.run.smithi129.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-29T10:55:18.804 INFO:teuthology.orchestra.run.smithi129.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 154 kB/s | 43 kB 00:00 2024-03-29T10:55:18.888 INFO:teuthology.orchestra.run.smithi129.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-29T10:55:18.938 INFO:teuthology.orchestra.run.smithi129.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-03-29T10:55:18.972 INFO:teuthology.orchestra.run.smithi129.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-03-29T10:55:19.006 INFO:teuthology.orchestra.run.smithi129.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 467 kB/s | 172 kB 00:00 2024-03-29T10:55:19.031 INFO:teuthology.orchestra.run.smithi129.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2024-03-29T10:55:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:55:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:19.035 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:55:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:55:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:55:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:55:19.036 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:55:19.036 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:19.036 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:55:19.056 INFO:teuthology.orchestra.run.smithi129.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-03-29T10:55:19.081 INFO:teuthology.orchestra.run.smithi129.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-29T10:55:19.115 INFO:teuthology.orchestra.run.smithi129.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-03-29T10:55:19.192 INFO:teuthology.orchestra.run.smithi129.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 212 kB/s | 16 kB 00:00 2024-03-29T10:55:19.217 INFO:teuthology.orchestra.run.smithi129.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 489 kB/s | 79 kB 00:00 2024-03-29T10:55:19.243 INFO:teuthology.orchestra.run.smithi129.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-03-29T10:55:19.268 INFO:teuthology.orchestra.run.smithi129.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T10:55:19.293 INFO:teuthology.orchestra.run.smithi129.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 472 kB/s | 36 kB 00:00 2024-03-29T10:55:19.344 INFO:teuthology.orchestra.run.smithi129.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-29T10:55:19.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2024-03-29T10:55:19.377 INFO:teuthology.orchestra.run.smithi129.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-29T10:55:19.394 INFO:teuthology.orchestra.run.smithi129.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-29T10:55:19.401 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:19.401 INFO:teuthology.orchestra.run.smithi129.stdout:Total 8.8 MB/s | 30 MB 00:03 2024-03-29T10:55:19.426 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:55:19.468 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:55:19.468 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:55:19.903 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:55:19.903 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:55:19.918 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:55:20.079 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout:========================================================================================= 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout:========================================================================================= 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:55:20.082 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:55:20.083 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:55:20.084 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout:========================================================================================= 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout:Install 36 Packages 2024-03-29T10:55:20.085 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:20.086 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 30 M 2024-03-29T10:55:20.086 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 107 M 2024-03-29T10:55:20.086 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:55:20.589 INFO:teuthology.orchestra.run.smithi107.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-29T10:55:20.714 INFO:teuthology.orchestra.run.smithi107.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.9 MB/s | 2.1 MB 00:00 2024-03-29T10:55:21.048 INFO:teuthology.orchestra.run.smithi107.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-03-29T10:55:21.165 INFO:teuthology.orchestra.run.smithi107.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.1 MB/s | 247 kB 00:00 2024-03-29T10:55:21.262 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:55:21.290 INFO:teuthology.orchestra.run.smithi107.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T10:55:21.314 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:55:21.332 INFO:teuthology.orchestra.run.smithi107.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:55:21.347 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:55:21.366 INFO:teuthology.orchestra.run.smithi107.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2024-03-29T10:55:21.374 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:55:21.399 INFO:teuthology.orchestra.run.smithi107.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:55:21.445 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:55:21.498 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:55:21.531 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:55:21.575 INFO:teuthology.orchestra.run.smithi107.stdout:(9/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-29T10:55:21.600 INFO:teuthology.orchestra.run.smithi107.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 193 kB/s | 39 kB 00:00 2024-03-29T10:55:21.622 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:55:21.625 INFO:teuthology.orchestra.run.smithi107.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:55:21.684 INFO:teuthology.orchestra.run.smithi107.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T10:55:21.693 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:55:21.709 INFO:teuthology.orchestra.run.smithi107.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T10:55:21.734 INFO:teuthology.orchestra.run.smithi107.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:55:21.743 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:55:21.759 INFO:teuthology.orchestra.run.smithi107.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-29T10:55:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:55:21.843 INFO:teuthology.orchestra.run.smithi107.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-29T10:55:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:55:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:55:21.918 INFO:teuthology.orchestra.run.smithi107.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 681 kB/s | 125 kB 00:00 2024-03-29T10:55:21.956 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:55:22.020 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:55:22.027 INFO:teuthology.orchestra.run.smithi107.stdout:(18/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-03-29T10:55:22.061 INFO:teuthology.orchestra.run.smithi107.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 306 kB/s | 43 kB 00:00 2024-03-29T10:55:22.094 INFO:teuthology.orchestra.run.smithi107.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 602 kB/s | 151 kB 00:00 2024-03-29T10:55:22.153 INFO:teuthology.orchestra.run.smithi107.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-03-29T10:55:22.205 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:55:22.220 INFO:teuthology.orchestra.run.smithi107.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 266 kB/s | 18 kB 00:00 2024-03-29T10:55:22.245 INFO:teuthology.orchestra.run.smithi107.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-03-29T10:55:22.253 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:55:22.279 INFO:teuthology.orchestra.run.smithi107.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 684 kB/s | 172 kB 00:00 2024-03-29T10:55:22.292 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:55:22.304 INFO:teuthology.orchestra.run.smithi107.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 276 kB/s | 23 kB 00:00 2024-03-29T10:55:22.329 INFO:teuthology.orchestra.run.smithi107.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-03-29T10:55:22.334 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:55:22.354 INFO:teuthology.orchestra.run.smithi107.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-29T10:55:22.365 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:55:22.388 INFO:teuthology.orchestra.run.smithi107.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-29T10:55:22.421 INFO:teuthology.orchestra.run.smithi107.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 503 kB/s | 46 kB 00:00 2024-03-29T10:55:22.441 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:55:22.447 INFO:teuthology.orchestra.run.smithi107.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-29T10:55:22.472 INFO:teuthology.orchestra.run.smithi107.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-29T10:55:22.475 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:55:22.505 INFO:teuthology.orchestra.run.smithi107.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-03-29T10:55:22.508 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:55:22.539 INFO:teuthology.orchestra.run.smithi107.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-29T10:55:22.549 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:55:22.564 INFO:teuthology.orchestra.run.smithi107.stdout:(34/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 630 kB/s | 90 kB 00:00 2024-03-29T10:55:22.584 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:55:22.598 INFO:teuthology.orchestra.run.smithi107.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-29T10:55:22.619 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:55:22.623 INFO:teuthology.orchestra.run.smithi107.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-29T10:55:22.626 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:22.626 INFO:teuthology.orchestra.run.smithi107.stdout:Total 12 MB/s | 30 MB 00:02 2024-03-29T10:55:22.651 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:55:22.693 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:55:22.693 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:55:22.700 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:55:22.741 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:55:22.814 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:55:22.952 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:55:23.086 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:55:23.122 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:55:23.123 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:55:23.242 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:55:23.272 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:55:23.297 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:55:23.297 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:23.297 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:55:23.297 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:55:23.297 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:55:23.298 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:24.442 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:55:24.509 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:55:24.559 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:55:24.586 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:55:24.658 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:55:24.693 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:55:24.726 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:55:24.731 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:55:24.757 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:55:24.758 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:24.758 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:55:24.758 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:55:24.758 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:55:24.758 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:24.818 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:55:24.887 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:55:24.937 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:55:25.009 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:55:25.063 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:55:25.101 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:55:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:55:25.207 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:55:25.214 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:55:25.230 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:55:25.230 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:25.230 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:55:25.230 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:55:25.230 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:55:25.230 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:25.405 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:55:25.441 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:55:25.455 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:55:25.463 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:55:25.463 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:25.464 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:55:25.464 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:55:25.464 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:55:25.464 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:25.485 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:55:25.520 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:55:25.530 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:55:25.550 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:55:25.626 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:55:25.668 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:55:25.701 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:55:25.741 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:55:25.776 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:55:25.803 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:55:25.893 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:55:25.928 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:55:25.999 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:55:26.135 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:55:26.252 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:55:26.411 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:55:26.441 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:55:26.467 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:55:26.468 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:26.468 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:55:26.468 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:55:26.468 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:55:26.468 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:27.280 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:55:27.280 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:55:27.280 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:55:27.280 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:55:27.280 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:55:27.280 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:55:27.281 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:55:27.282 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:55:27.284 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:55:27.901 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:55:27.927 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:55:27.927 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:27.927 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:55:27.928 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:55:27.928 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:55:27.928 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:28.218 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:55:28.218 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:28.218 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:55:28.218 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:28.218 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:28.218 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:55:28.219 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:55:28.220 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:55:28.221 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:55:28.221 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:55:28.221 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:55:28.221 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:55:28.221 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:55:28.221 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:55:28.221 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:55:28.221 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:28.221 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:28.377 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:55:28.400 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:55:28.400 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:28.400 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:55:28.400 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:55:28.400 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:55:28.400 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:28.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-base 2024-03-29T10:55:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:55:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:55:28.642 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:28.643 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:55:28.643 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:55:28.643 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:55:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:28.709 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:55:29.106 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:29.205 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:55:29.258 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:29.259 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-29T10:55:29.259 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:29.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install cephadm 2024-03-29T10:55:29.886 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:30.037 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:30.037 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:55:30.037 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:55:30.037 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:55:30.037 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:55:30.038 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:55:30.038 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:30.038 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:55:30.038 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:55:30.038 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-29T10:55:30.038 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:30.038 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 747 k 2024-03-29T10:55:30.039 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 754 k 2024-03-29T10:55:30.039 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:55:30.324 INFO:teuthology.orchestra.run.smithi129.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T10:55:30.324 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:30.324 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-29T10:55:30.325 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:55:30.330 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:55:30.330 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:55:30.335 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:55:30.336 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:55:30.409 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:55:30.485 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:55:30.486 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:55:30.487 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:55:30.774 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:55:30.870 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:55:31.262 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:55:31.555 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:55:31.555 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:31.555 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:55:31.555 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:31.555 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:31.555 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:31.594 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:55:31.594 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:31.594 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:55:31.594 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:55:31.595 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:55:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:55:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:55:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:55:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:55:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:55:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:55:31.596 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:55:31.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:55:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:55:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:55:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:55:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:55:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:55:31.598 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:31.598 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:55:31.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:55:31.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2024-03-29T10:55:32.327 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:32.480 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:32.480 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:32.481 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 145 k 2024-03-29T10:55:32.482 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 436 k 2024-03-29T10:55:32.482 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:55:32.538 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:55:32.636 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:55:32.687 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:55:32.688 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-29T10:55:32.688 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:55:32.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2024-03-29T10:55:32.831 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 416 kB/s | 145 kB 00:00 2024-03-29T10:55:32.831 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:32.831 INFO:teuthology.orchestra.run.smithi129.stdout:Total 415 kB/s | 145 kB 00:00 2024-03-29T10:55:32.832 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:55:32.839 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:55:32.839 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:55:32.891 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:55:32.892 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:55:33.031 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:55:33.090 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:55:33.118 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:55:33.118 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:33.118 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:55:33.118 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:33.318 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:55:33.469 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:55:33.469 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 747 k 2024-03-29T10:55:33.470 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 754 k 2024-03-29T10:55:33.471 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:55:33.941 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.6 MB/s | 747 kB 00:00 2024-03-29T10:55:33.942 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:33.942 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.5 MB/s | 747 kB 00:00 2024-03-29T10:55:33.942 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:55:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:55:33.947 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:55:33.953 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:55:33.953 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:55:34.025 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:55:34.062 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:55:34.062 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:34.062 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:55:34.062 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:34.062 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:34.062 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:34.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr 2024-03-29T10:55:34.432 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:55:34.521 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:55:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:55:34.882 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:34.979 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:55:35.033 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:35.034 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-29T10:55:35.034 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:35.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:55:35.170 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:55:35.171 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:35.171 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:55:35.171 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:35.171 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:35.171 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:55:35.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:55:35.665 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:35.818 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:35.819 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2024-03-29T10:55:35.819 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:55:35.819 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2024-03-29T10:55:35.819 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:55:35.819 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:55:35.819 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:55:35.819 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:55:35.819 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2024-03-29T10:55:35.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:35.821 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 6.9 M 2024-03-29T10:55:35.821 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 26 M 2024-03-29T10:55:35.821 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:55:35.975 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:55:36.131 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-29T10:55:36.132 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:36.133 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 145 k 2024-03-29T10:55:36.133 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 436 k 2024-03-29T10:55:36.133 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:55:36.228 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 165 kB/s | 31 kB 00:00 2024-03-29T10:55:36.370 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 46 kB/s | 15 kB 00:00 2024-03-29T10:55:36.454 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-29T10:55:36.479 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 238 kB 00:00 2024-03-29T10:55:36.506 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 390 kB/s | 145 kB 00:00 2024-03-29T10:55:36.507 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:36.507 INFO:teuthology.orchestra.run.smithi107.stdout:Total 388 kB/s | 145 kB 00:00 2024-03-29T10:55:36.507 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:55:36.514 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:55:36.515 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:55:36.568 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:55:36.568 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:55:36.705 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:55:36.737 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T10:55:36.765 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:55:36.790 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:55:36.790 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:55:36.790 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:55:36.791 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:36.796 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-29T10:55:36.830 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-03-29T10:55:36.863 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.2 MB/s | 1.8 MB 00:00 2024-03-29T10:55:37.064 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.6 MB/s | 2.0 MB 00:00 2024-03-29T10:55:37.131 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T10:55:37.761 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:55:37.761 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:37.762 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:55:37.762 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:55:37.762 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:37.763 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:55:38.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2024-03-29T10:55:38.591 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:55:38.690 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:55:38.742 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:55:38.743 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-29T10:55:38.743 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:55:38.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:55:39.377 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:55:39.527 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:55:39.528 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2024-03-29T10:55:39.528 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:55:39.528 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2024-03-29T10:55:39.528 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:55:39.529 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:55:39.530 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:55:39.530 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:39.530 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:55:39.530 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2024-03-29T10:55:39.530 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2024-03-29T10:55:39.530 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:39.530 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 6.9 M 2024-03-29T10:55:39.531 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 26 M 2024-03-29T10:55:39.531 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:55:39.895 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 80 kB/s | 15 kB 00:00 2024-03-29T10:55:39.953 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 126 kB/s | 31 kB 00:00 2024-03-29T10:55:40.039 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 9.7 kB/s | 31 kB 00:03 2024-03-29T10:55:40.086 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-29T10:55:40.097 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 44 kB/s | 144 kB 00:03 2024-03-29T10:55:40.101 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:40.101 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.6 MB/s | 6.9 MB 00:04 2024-03-29T10:55:40.121 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:55:40.140 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:55:40.140 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:55:40.187 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 8.4 MB/s | 863 kB 00:00 2024-03-29T10:55:40.229 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-03-29T10:55:40.305 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:55:40.306 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:55:40.354 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-03-29T10:55:40.496 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.2 MB/s | 1.8 MB 00:00 2024-03-29T10:55:40.521 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T10:55:40.580 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 525 kB/s | 31 kB 00:00 2024-03-29T10:55:40.647 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T10:55:40.697 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.9 MB/s | 2.0 MB 00:00 2024-03-29T10:55:40.722 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 639 kB/s | 144 kB 00:00 2024-03-29T10:55:40.726 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:40.726 INFO:teuthology.orchestra.run.smithi107.stdout:Total 5.7 MB/s | 6.9 MB 00:01 2024-03-29T10:55:40.744 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:55:40.823 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:55:40.881 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:55:40.917 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:55:40.949 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:55:40.949 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:55:40.963 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:55:41.010 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:55:41.110 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:55:41.234 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:55:41.322 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:55:41.373 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:55:41.402 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:55:41.433 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:55:41.462 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:55:41.539 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:55:41.583 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:55:41.637 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:55:41.674 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:55:41.677 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:55:41.700 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:55:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:55:41.911 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:55:42.004 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:55:42.057 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:55:42.085 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:55:42.125 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:55:42.375 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:55:42.409 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:55:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:55:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:55:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:55:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:55:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:55:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:55:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:55:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:55:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:55:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:55:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:55:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:55:42.811 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:55:42.811 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:42.811 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:55:42.812 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:42.813 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:43.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:55:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:55:43.223 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:55:43.588 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:55:43.589 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:55:43.589 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:55:43.589 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:55:43.589 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:43.589 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:55:43.637 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:43.789 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:55:43.790 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2024-03-29T10:55:43.791 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:43.792 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 37 M 2024-03-29T10:55:43.792 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 209 M 2024-03-29T10:55:43.792 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:55:43.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:55:44.330 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 253 kB/s | 32 kB 00:00 2024-03-29T10:55:44.371 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 424 kB/s | 17 kB 00:00 2024-03-29T10:55:44.421 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:55:44.438 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T10:55:44.464 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 381 kB/s | 34 kB 00:00 2024-03-29T10:55:44.575 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:55:44.576 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-29T10:55:44.576 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:55:44.576 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-29T10:55:44.576 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:55:44.576 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:55:44.576 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2024-03-29T10:55:44.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:44.578 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 37 M 2024-03-29T10:55:44.578 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 209 M 2024-03-29T10:55:44.578 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:55:44.639 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-03-29T10:55:44.706 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T10:55:44.731 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:55:44.830 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 216 kB/s | 32 kB 00:00 2024-03-29T10:55:44.880 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 349 kB/s | 17 kB 00:00 2024-03-29T10:55:44.922 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 823 kB/s | 34 kB 00:00 2024-03-29T10:55:44.997 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.5 MB/s | 3.0 MB 00:00 2024-03-29T10:55:45.048 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-29T10:55:45.082 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-29T10:55:45.140 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-03-29T10:55:45.174 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-29T10:55:45.264 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-03-29T10:55:45.339 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2024-03-29T10:55:45.365 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T10:55:45.698 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-29T10:55:45.866 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.4 MB/s | 7.4 MB 00:01 2024-03-29T10:55:45.869 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:45.869 INFO:teuthology.orchestra.run.smithi129.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-29T10:55:45.910 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:55:45.934 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:55:45.934 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:55:46.304 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:55:46.304 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:55:46.424 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 4.2 MB/s | 7.4 MB 00:01 2024-03-29T10:55:46.449 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 260 kB/s | 195 kB 00:00 2024-03-29T10:55:46.666 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-29T10:55:47.003 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:55:47.080 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:55:47.112 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:55:47.182 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 442 kB/s | 803 kB 00:01 2024-03-29T10:55:47.186 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:47.187 INFO:teuthology.orchestra.run.smithi107.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-29T10:55:47.203 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:55:47.230 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:55:47.242 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:55:47.255 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:55:47.255 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:55:47.416 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:55:47.464 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:55:47.545 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:55:47.616 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:55:47.616 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:55:48.201 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:55:48.301 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:55:48.325 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:55:48.406 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:55:48.445 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:55:48.549 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:55:48.590 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:55:48.767 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:55:48.798 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:55:48.881 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:55:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:55:49.530 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:55:49.637 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:55:50.538 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:55:50.566 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:55:50.874 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:55:51.908 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:55:51.939 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:55:53.109 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:55:53.109 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:55:53.109 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:55:53.109 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:55:53.109 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:55:53.109 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:55:53.109 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:55:53.110 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:55:53.110 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:55:53.110 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:55:53.110 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:55:53.110 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:55:53.521 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:55:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:53.523 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:53.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:55:54.460 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:54.611 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:54.611 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-29T10:55:54.612 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:55:54.612 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-29T10:55:54.612 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:55:54.612 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:55:54.612 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:55:54.612 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:55:54.613 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:55:54.613 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:55:54.613 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:55:54.613 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:55:54.613 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:55:54.613 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:55:54.614 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:55:54.614 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:55:54.614 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:55:54.614 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:55:54.614 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:55:54.614 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:54.615 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:55:54.615 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-29T10:55:54.615 INFO:teuthology.orchestra.run.smithi129.stdout:Install 13 Packages 2024-03-29T10:55:54.615 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:54.616 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.0 M 2024-03-29T10:55:54.616 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 26 M 2024-03-29T10:55:54.616 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:55:54.710 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:55:54.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:55:54.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:55:54.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:55:54.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:55:54.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:55:54.710 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:55:54.711 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:55:54.711 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:55:54.711 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:55:54.711 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:55:54.711 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:55:55.093 INFO:teuthology.orchestra.run.smithi129.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 194 kB/s | 24 kB 00:00 2024-03-29T10:55:55.110 INFO:teuthology.orchestra.run.smithi129.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:55:55.132 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:55:55.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:55:55.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:55:55.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:55:55.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:55.133 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:55:55.160 INFO:teuthology.orchestra.run.smithi129.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T10:55:55.211 INFO:teuthology.orchestra.run.smithi129.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-29T10:55:55.236 INFO:teuthology.orchestra.run.smithi129.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-29T10:55:55.253 INFO:teuthology.orchestra.run.smithi129.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T10:55:55.270 INFO:teuthology.orchestra.run.smithi129.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 173 kB/s | 52 kB 00:00 2024-03-29T10:55:55.421 INFO:teuthology.orchestra.run.smithi129.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 90 kB/s | 15 kB 00:00 2024-03-29T10:55:55.437 INFO:teuthology.orchestra.run.smithi129.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 160 kB/s | 32 kB 00:00 2024-03-29T10:55:55.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:55:55.546 INFO:teuthology.orchestra.run.smithi129.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-03-29T10:55:55.571 INFO:teuthology.orchestra.run.smithi129.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 673 kB/s | 203 kB 00:00 2024-03-29T10:55:55.613 INFO:teuthology.orchestra.run.smithi129.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-29T10:55:55.755 INFO:teuthology.orchestra.run.smithi129.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-29T10:55:55.757 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:55.757 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-29T10:55:55.757 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:55:55.770 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:55:55.770 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:55:55.861 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:55:55.862 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:55:56.082 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:55:56.234 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:55:56.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-03-29T10:55:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:55:56.237 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.0 M 2024-03-29T10:55:56.237 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 26 M 2024-03-29T10:55:56.237 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:55:56.364 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:55:56.543 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:55:56.559 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 195 kB/s | 24 kB 00:00 2024-03-29T10:55:56.584 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2024-03-29T10:55:56.589 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:55:56.643 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-29T10:55:56.644 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:55:56.668 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-29T10:55:56.673 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:55:56.693 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:55:56.714 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:55:56.719 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-29T10:55:56.744 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 168 kB/s | 52 kB 00:00 2024-03-29T10:55:56.817 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:55:56.886 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-29T10:55:56.911 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-03-29T10:55:56.917 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:55:56.984 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:55:57.036 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-03-29T10:55:57.062 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 638 kB/s | 203 kB 00:00 2024-03-29T10:55:57.162 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-03-29T10:55:57.196 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-03-29T10:55:57.197 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:55:57.197 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-29T10:55:57.198 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:55:57.211 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:55:57.211 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:55:57.300 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:55:57.301 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:55:57.694 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:55:57.756 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:55:57.790 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:55:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:55:57.839 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:55:57.868 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:55:57.974 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:55:58.038 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:55:58.093 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:55:58.140 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:55:58.189 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:55:58.292 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:55:58.401 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:55:58.458 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:55:58.608 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:55:58.608 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:55:58.608 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:55:58.608 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:55:58.608 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:55:58.608 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:55:58.608 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:55:58.608 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:55:58.608 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:55:58.609 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:55:58.609 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:55:58.609 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:55:58.609 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:55:58.959 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:55:58.959 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:58.959 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:55:58.960 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:58.961 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:55:59.174 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:55:59.222 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:55:59.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:55:59.256 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:55:59.304 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:55:59.326 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:55:59.818 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:24 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:55:59.972 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 6.8 M 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 33 M 2024-03-29T10:55:59.973 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:56:00.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:56:00.101 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:56:00.101 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:56:00.101 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:56:00.375 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T10:56:00.434 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 775 kB/s | 248 kB 00:00 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:56:00.470 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:56:00.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:56:00.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:56:00.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:56:00.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:56:00.471 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:56:00.471 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:00.471 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:00.509 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 349 kB/s | 138 kB 00:00 2024-03-29T10:56:00.526 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 627 kB/s | 58 kB 00:00 2024-03-29T10:56:00.560 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.9 MB/s | 548 kB 00:00 2024-03-29T10:56:00.602 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 551 kB/s | 51 kB 00:00 2024-03-29T10:56:00.605 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:00.605 INFO:teuthology.orchestra.run.smithi129.stdout:Total 11 MB/s | 6.8 MB 00:00 2024-03-29T10:56:00.618 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:56:00.628 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:56:00.628 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:56:00.716 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:56:00.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:56:00.718 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:56:01.016 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:56:01.185 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:56:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:56:01.285 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:01.438 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:01.438 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-29T10:56:01.438 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:56:01.439 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:01.440 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:56:01.440 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================== 2024-03-29T10:56:01.440 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-03-29T10:56:01.440 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:01.440 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 6.8 M 2024-03-29T10:56:01.441 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 33 M 2024-03-29T10:56:01.441 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:56:01.765 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:56:01.839 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:56:01.900 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:56:01.925 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:56:02.132 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T10:56:02.158 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 469 kB/s | 138 kB 00:00 2024-03-29T10:56:02.216 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 994 kB/s | 58 kB 00:00 2024-03-29T10:56:02.258 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-03-29T10:56:02.283 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.6 MB/s | 548 kB 00:00 2024-03-29T10:56:02.384 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 477 kB/s | 248 kB 00:00 2024-03-29T10:56:02.384 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:02.385 INFO:teuthology.orchestra.run.smithi107.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2024-03-29T10:56:02.398 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:56:02.408 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:56:02.408 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:56:02.496 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:56:02.497 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:56:02.665 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:56:02.666 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:56:02.666 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:56:02.666 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:56:02.666 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:56:02.666 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:56:02.801 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:56:02.958 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:56:02.973 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:56:02.973 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:02.974 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:56:02.974 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:56:02.974 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:56:02.974 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:56:02.974 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:56:02.974 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:56:02.974 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:56:02.974 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:02.974 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:03.011 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:56:03.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-fuse 2024-03-29T10:56:03.550 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:56:03.631 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:56:03.691 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:56:03.714 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:56:03.768 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:03.918 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2024-03-29T10:56:03.919 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:03.920 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 964 k 2024-03-29T10:56:03.920 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 2.8 M 2024-03-29T10:56:03.920 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:56:04.046 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 625 kB/s | 79 kB 00:00 2024-03-29T10:56:04.221 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.9 MB/s | 885 kB 00:00 2024-03-29T10:56:04.222 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:04.223 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.1 MB/s | 964 kB 00:00 2024-03-29T10:56:04.223 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:56:04.242 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:56:04.242 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:56:04.315 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:56:04.316 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:56:04.498 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:56:04.522 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:56:04.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:56:04.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:56:04.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:56:04.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:56:04.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:56:04.642 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:56:04.704 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:04.842 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:05.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2024-03-29T10:56:05.388 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:56:05.388 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:56:05.644 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:05.666 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:56:05.666 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:05.666 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:56:05.666 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:56:05.666 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:05.667 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:05.793 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:05.793 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:05.793 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:56:05.793 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:05.793 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:56:05.793 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:56:05.794 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:56:05.794 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:56:05.794 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:05.794 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:56:05.794 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:05.794 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-03-29T10:56:05.794 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:05.794 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 964 k 2024-03-29T10:56:05.795 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.8 M 2024-03-29T10:56:05.795 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:56:05.919 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 633 kB/s | 79 kB 00:00 2024-03-29T10:56:05.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-volume 2024-03-29T10:56:06.236 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 885 kB 00:00 2024-03-29T10:56:06.238 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:06.238 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-29T10:56:06.238 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:56:06.258 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:56:06.258 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:56:06.332 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:56:06.332 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:56:06.509 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:56:06.551 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:06.671 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:56:06.710 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:06.710 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:06.710 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2024-03-29T10:56:06.711 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:06.712 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 332 k 2024-03-29T10:56:06.712 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 1.5 M 2024-03-29T10:56:06.712 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:56:06.745 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:56:06.836 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 555 kB/s | 69 kB 00:00 2024-03-29T10:56:07.111 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 661 kB/s | 264 kB 00:00 2024-03-29T10:56:07.112 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:07.113 INFO:teuthology.orchestra.run.smithi129.stdout:Total 828 kB/s | 332 kB 00:00 2024-03-29T10:56:07.113 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:56:07.121 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:56:07.121 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:56:07.157 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:56:07.157 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:56:07.306 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:56:07.364 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:56:07.364 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:56:07.484 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:56:07.531 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:56:07.547 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:56:07.548 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:56:07.548 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:56:07.548 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:07.681 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:56:07.681 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:07.682 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:56:07.682 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:56:07.682 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:07.682 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:08.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-volume 2024-03-29T10:56:08.305 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:56:08.586 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:08.614 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:56:08.614 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:08.614 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:56:08.614 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:56:08.614 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:56:08.615 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:08.615 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:08.738 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:08.738 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-03-29T10:56:08.739 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:08.740 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 332 k 2024-03-29T10:56:08.740 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.5 M 2024-03-29T10:56:08.740 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:56:08.884 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 478 kB/s | 69 kB 00:00 2024-03-29T10:56:08.953 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados-devel 2024-03-29T10:56:09.192 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 583 kB/s | 264 kB 00:00 2024-03-29T10:56:09.193 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:09.194 INFO:teuthology.orchestra.run.smithi107.stdout:Total 731 kB/s | 332 kB 00:00 2024-03-29T10:56:09.194 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:56:09.202 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:56:09.202 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:56:09.241 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:56:09.242 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:56:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:56:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:09.576 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:56:09.612 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:56:09.628 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:56:09.628 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:56:09.628 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:56:09.628 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:09.681 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:09.682 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:09.682 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-03-29T10:56:09.682 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:09.682 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:56:09.682 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:56:09.682 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:09.682 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:56:09.682 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:09.683 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-29T10:56:09.683 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:09.683 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 127 k 2024-03-29T10:56:09.683 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 456 k 2024-03-29T10:56:09.683 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:56:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 541 kB/s | 127 kB 00:00 2024-03-29T10:56:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:Total 537 kB/s | 127 kB 00:00 2024-03-29T10:56:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:56:09.924 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:56:09.924 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:56:09.980 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:56:09.980 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:56:10.120 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:56:10.226 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:10.352 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:56:10.654 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:56:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:10.687 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:56:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:56:10.687 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:56:10.688 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:10.688 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:10.955 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:10.956 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:10.956 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:56:10.956 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:10.956 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:10.956 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:11.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2024-03-29T10:56:11.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs2 2024-03-29T10:56:11.621 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:11.773 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:11.773 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:11.773 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-29T10:56:11.773 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:11.773 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:56:11.774 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:56:11.774 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:11.774 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:56:11.774 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:11.774 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-29T10:56:11.774 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:11.774 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 127 k 2024-03-29T10:56:11.775 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 456 k 2024-03-29T10:56:11.775 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:56:11.779 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:11.880 INFO:teuthology.orchestra.run.smithi129.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:11.933 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:11.934 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-29T10:56:11.934 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:12.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs-devel 2024-03-29T10:56:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 391 kB/s | 127 kB 00:00 2024-03-29T10:56:12.100 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:12.100 INFO:teuthology.orchestra.run.smithi107.stdout:Total 389 kB/s | 127 kB 00:00 2024-03-29T10:56:12.100 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:56:12.105 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:56:12.105 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:56:12.163 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:56:12.163 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:56:12.315 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:56:12.414 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:12.574 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:12.726 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:12.727 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:12.727 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-03-29T10:56:12.727 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:12.727 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:56:12.727 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:56:12.727 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:12.727 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:56:12.727 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:12.728 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-29T10:56:12.728 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:12.728 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 31 k 2024-03-29T10:56:12.728 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 140 k 2024-03-29T10:56:12.728 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:56:12.858 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:12.966 INFO:teuthology.orchestra.run.smithi129.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 131 kB/s | 31 kB 00:00 2024-03-29T10:56:12.966 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:12.966 INFO:teuthology.orchestra.run.smithi129.stdout:Total 130 kB/s | 31 kB 00:00 2024-03-29T10:56:12.967 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:56:12.970 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:56:12.970 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:56:12.993 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:56:12.994 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:56:13.120 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:56:13.156 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:13.156 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:56:13.156 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:13.156 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:13.156 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:13.213 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:13.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2024-03-29T10:56:13.593 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:13.865 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:13.865 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:13.865 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:56:13.865 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:13.865 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:13.865 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:13.971 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:14.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados2 2024-03-29T10:56:14.073 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:14.128 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:14.129 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-29T10:56:14.129 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:14.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2024-03-29T10:56:14.579 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:14.678 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:14.729 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:14.730 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-29T10:56:14.730 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:14.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librbd1 2024-03-29T10:56:14.778 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:14.929 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:14.930 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:14.930 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-29T10:56:14.930 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:14.930 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:56:14.930 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:56:14.930 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:14.930 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:56:14.931 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:14.931 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-29T10:56:14.931 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:14.931 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 31 k 2024-03-29T10:56:14.931 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 140 k 2024-03-29T10:56:14.931 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:56:15.171 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 130 kB/s | 31 kB 00:00 2024-03-29T10:56:15.172 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:15.172 INFO:teuthology.orchestra.run.smithi107.stdout:Total 129 kB/s | 31 kB 00:00 2024-03-29T10:56:15.172 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:56:15.175 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:56:15.176 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:56:15.208 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:56:15.209 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:56:15.334 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:56:15.340 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:15.428 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:15.438 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:15.490 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:15.490 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-29T10:56:15.491 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:15.538 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rados 2024-03-29T10:56:15.795 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:16.080 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:16.080 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:16.080 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:56:16.080 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:16.080 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:16.080 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:16.105 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:16.202 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:16.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2024-03-29T10:56:16.254 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:16.255 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-29T10:56:16.255 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:16.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rgw 2024-03-29T10:56:16.812 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:16.879 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:16.910 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:16.961 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:16.962 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-29T10:56:16.962 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:16.977 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:17.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2024-03-29T10:56:17.029 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:17.030 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-29T10:56:17.030 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:17.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-cephfs 2024-03-29T10:56:17.579 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:17.648 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:17.676 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:17.727 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:17.728 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-29T10:56:17.728 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:17.746 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:17.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2024-03-29T10:56:17.798 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:17.799 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-29T10:56:17.799 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:17.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rbd 2024-03-29T10:56:18.354 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:18.414 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:18.452 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:18.504 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:18.505 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-29T10:56:18.505 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:18.512 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:18.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2024-03-29T10:56:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-29T10:56:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:18.613 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-fuse 2024-03-29T10:56:19.126 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:19.180 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:19.225 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:19.276 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:19.278 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-29T10:56:19.278 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:19.329 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:19.329 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:19.329 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-29T10:56:19.329 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:19.329 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:56:19.330 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:56:19.330 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:19.330 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:56:19.330 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:19.330 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-29T10:56:19.330 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:19.330 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 85 k 2024-03-29T10:56:19.331 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 231 k 2024-03-29T10:56:19.331 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:56:19.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2024-03-29T10:56:19.645 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 272 kB/s | 85 kB 00:00 2024-03-29T10:56:19.645 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:19.645 INFO:teuthology.orchestra.run.smithi129.stdout:Total 271 kB/s | 85 kB 00:00 2024-03-29T10:56:19.645 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:56:19.651 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:56:19.652 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:56:19.707 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:56:19.707 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:56:19.856 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:56:19.895 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:19.959 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:19.992 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:20.043 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:20.044 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-29T10:56:20.044 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:20.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2024-03-29T10:56:20.349 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:20.647 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:20.647 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:20.647 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:56:20.647 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:20.647 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:20.647 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:20.663 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:20.761 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:56:20.812 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:20.813 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-03-29T10:56:20.813 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:20.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-mirror 2024-03-29T10:56:20.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2024-03-29T10:56:21.437 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:21.465 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:21.584 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:21.585 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-03-29T10:56:21.585 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:21.585 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:56:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:56:21.585 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:21.585 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:56:21.586 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:21.586 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-29T10:56:21.586 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:21.586 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 85 k 2024-03-29T10:56:21.586 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 231 k 2024-03-29T10:56:21.586 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:56:21.625 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:21.626 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:21.626 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-03-29T10:56:21.626 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:21.626 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:56:21.626 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:56:21.626 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:21.626 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:56:21.626 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:21.627 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-29T10:56:21.627 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:21.627 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 3.1 M 2024-03-29T10:56:21.627 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 13 M 2024-03-29T10:56:21.627 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:56:21.808 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 385 kB/s | 85 kB 00:00 2024-03-29T10:56:21.809 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:21.809 INFO:teuthology.orchestra.run.smithi107.stdout:Total 383 kB/s | 85 kB 00:00 2024-03-29T10:56:21.809 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:56:21.816 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:56:21.816 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:56:21.872 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:56:21.872 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:56:22.009 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:56:22.106 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:22.538 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:22.607 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.2 MB/s | 3.1 MB 00:00 2024-03-29T10:56:22.607 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:22.608 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.1 MB/s | 3.1 MB 00:00 2024-03-29T10:56:22.608 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:56:22.617 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:56:22.617 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:56:22.691 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:56:22.691 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:56:22.863 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:22.865 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:22.865 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:56:22.865 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:22.865 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:22.865 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:23.116 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:56:23.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2024-03-29T10:56:23.183 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:23.208 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:23.208 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:56:23.208 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:56:23.208 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:56:23.208 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:56:23.208 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:23.710 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:23.867 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.1 M 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 13 M 2024-03-29T10:56:23.868 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:56:24.472 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:24.472 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:24.472 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:56:24.472 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:24.472 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:24.472 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:24.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-nbd 2024-03-29T10:56:24.770 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.4 MB/s | 3.1 MB 00:00 2024-03-29T10:56:24.771 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:24.771 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.4 MB/s | 3.1 MB 00:00 2024-03-29T10:56:24.771 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:56:24.782 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:56:24.782 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:56:24.856 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:56:24.856 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:56:25.282 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:56:25.310 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:54:35 AM UTC. 2024-03-29T10:56:25.346 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:25.371 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:25.372 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:56:25.372 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:56:25.372 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:56:25.372 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:56:25.372 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:25.458 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-29T10:56:25.459 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:25.459 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-29T10:56:25.459 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:25.459 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-29T10:56:25.459 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:56:25.460 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:25.460 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-29T10:56:25.460 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-29T10:56:25.460 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-29T10:56:25.460 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:25.460 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 171 k 2024-03-29T10:56:25.460 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 494 k 2024-03-29T10:56:25.460 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-29T10:56:25.662 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 851 kB/s | 171 kB 00:00 2024-03-29T10:56:25.662 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:25.662 INFO:teuthology.orchestra.run.smithi129.stdout:Total 846 kB/s | 171 kB 00:00 2024-03-29T10:56:25.662 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-29T10:56:25.670 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-29T10:56:25.670 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-29T10:56:25.726 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-29T10:56:25.726 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-29T10:56:25.872 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-29T10:56:25.994 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:26.417 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:26.604 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:26.604 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:26.604 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:56:26.604 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:26.604 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:26.605 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:26.729 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:26.729 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:26.729 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-29T10:56:26.729 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:26.729 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:26.729 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-29T10:56:26.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-03-29T10:56:26.974 DEBUG:teuthology.parallel:result is None 2024-03-29T10:56:27.426 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:54:39 AM UTC. 2024-03-29T10:56:27.574 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-29T10:56:27.574 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:27.574 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-03-29T10:56:27.574 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 171 k 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 494 k 2024-03-29T10:56:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-29T10:56:27.908 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 515 kB/s | 171 kB 00:00 2024-03-29T10:56:27.909 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:56:27.909 INFO:teuthology.orchestra.run.smithi107.stdout:Total 512 kB/s | 171 kB 00:00 2024-03-29T10:56:27.909 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-29T10:56:27.917 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-29T10:56:27.917 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-29T10:56:27.973 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-29T10:56:27.973 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-29T10:56:28.116 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-29T10:56:28.216 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:28.951 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:56:28.951 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:28.951 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-29T10:56:28.951 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:56:28.951 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:28.951 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-29T10:56:29.180 DEBUG:teuthology.parallel:result is None 2024-03-29T10:56:29.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:56:29.403 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:56:29.425 INFO:teuthology.orchestra.run.smithi107.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:56:29.426 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:56:29.426 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:56:29.428 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:56:29.650 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:56:29.675 INFO:teuthology.orchestra.run.smithi129.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:56:29.676 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:56:29.676 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:56:29.678 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:56:29.678 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:29.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:56:29.707 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:56:29.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:56:29.746 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:56:29.746 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:29.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:56:29.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:56:29.841 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:56:29.842 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:56:29.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:56:29.944 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:56:29.944 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:29.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:56:29.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:56:30.038 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:56:30.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:56:30.068 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:56:30.137 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:56:30.138 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:30.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:56:30.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:56:30.231 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:56:30.231 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:56:30.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:56:30.327 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:56:30.412 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:56:30.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2024-03-29T10:56:30.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2024-03-29T10:56:30.442 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:56:30.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:56:30.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:56:30.511 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:56:30.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:56:30.551 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:56:30.589 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:56:30.589 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:56:30.589 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622667', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFzZxd2kvuz30bPI7xAlhJL5tJ7HLRwgTx6Tm6UbCbL2tMstDFr0/oYOlXARa7OJEEd80SMqSR3NCkqnlii78Q=', 'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNHbCoWq3xgvmIDZlynuq9vF0SpeCEjLpTRbMDfDt5Q/x/9Wy9L9q5yC27U+4CL6J7Pr+MJ8ognYTedVfMR8So='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:56:30.589 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:56:30.618 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:56:30.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:56:30.673 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:56:30.700 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:30.700 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:56:30.751 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:56:30.751 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-03-29T10:56:30.806 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:56:30.806 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:56:30.806 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T10:56:30.806 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:56:30.806 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:56:30.806 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-29 10:56:25.632939712 +0000 2024-03-29T10:56:30.806 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-29 10:53:49.855914621 +0000 2024-03-29T10:56:30.807 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-29 10:53:49.855914621 +0000 2024-03-29T10:56:30.807 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-29 10:53:49.855914621 +0000 2024-03-29T10:56:30.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:56:30.871 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-29T10:56:30.872 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-29T10:56:30.872 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000205707 s, 2.5 MB/s 2024-03-29T10:56:30.873 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:56:30.929 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-03-29T10:56:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:56:30.983 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:56:30.983 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T10:56:30.983 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:56:30.983 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:56:30.983 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-29 10:56:25.632939712 +0000 2024-03-29T10:56:30.983 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-29 10:53:50.187908669 +0000 2024-03-29T10:56:30.983 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-29 10:53:50.187908669 +0000 2024-03-29T10:56:30.984 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-29 10:53:50.187908669 +0000 2024-03-29T10:56:30.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:56:31.048 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-29T10:56:31.048 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-29T10:56:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000236636 s, 2.2 MB/s 2024-03-29T10:56:31.050 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:56:31.106 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-03-29T10:56:31.160 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:56:31.160 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:56:31.160 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:56:31.160 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:56:31.160 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:56:31.161 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-29 10:56:25.632939712 +0000 2024-03-29T10:56:31.161 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-29 10:53:50.533902467 +0000 2024-03-29T10:56:31.161 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-29 10:53:50.533902467 +0000 2024-03-29T10:56:31.161 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-29 10:53:50.533902467 +0000 2024-03-29T10:56:31.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:56:31.225 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-29T10:56:31.226 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-29T10:56:31.226 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000182064 s, 2.8 MB/s 2024-03-29T10:56:31.227 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:56:31.283 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-03-29T10:56:31.337 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:56:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:56:31.338 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T10:56:31.338 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:56:31.338 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:56:31.338 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-29 10:56:25.632939712 +0000 2024-03-29T10:56:31.338 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-29 10:53:50.885896157 +0000 2024-03-29T10:56:31.338 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-29 10:53:50.885896157 +0000 2024-03-29T10:56:31.338 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-29 10:53:50.885896157 +0000 2024-03-29T10:56:31.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:56:31.402 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-29T10:56:31.402 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-29T10:56:31.402 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000187654 s, 2.7 MB/s 2024-03-29T10:56:31.404 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:56:31.459 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:31.459 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:56:31.459 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:56:31.476 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:56:31.476 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-03-29T10:56:31.532 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:56:31.532 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:56:31.533 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T10:56:31.533 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:56:31.533 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:56:31.533 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-29 10:56:23.488583270 +0000 2024-03-29T10:56:31.533 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-29 10:53:53.105320515 +0000 2024-03-29T10:56:31.533 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-29 10:53:53.105320515 +0000 2024-03-29T10:56:31.533 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-29 10:53:53.105320515 +0000 2024-03-29T10:56:31.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:56:31.601 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-29T10:56:31.601 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-29T10:56:31.601 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000229357 s, 2.2 MB/s 2024-03-29T10:56:31.603 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:56:31.659 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-03-29T10:56:31.714 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:56:31.714 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:56:31.714 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:56:31.714 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:56:31.714 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:56:31.714 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-29 10:56:23.488583270 +0000 2024-03-29T10:56:31.714 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-29 10:53:53.434313410 +0000 2024-03-29T10:56:31.714 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-29 10:53:53.434313410 +0000 2024-03-29T10:56:31.714 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-29 10:53:53.434313410 +0000 2024-03-29T10:56:31.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:56:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-29T10:56:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-29T10:56:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000182588 s, 2.8 MB/s 2024-03-29T10:56:31.781 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:56:31.837 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-03-29T10:56:31.892 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:56:31.892 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:56:31.892 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T10:56:31.892 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:56:31.892 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:56:31.892 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-29 10:56:23.489583248 +0000 2024-03-29T10:56:31.892 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-29 10:53:53.762306327 +0000 2024-03-29T10:56:31.892 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-29 10:53:53.762306327 +0000 2024-03-29T10:56:31.892 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-29 10:53:53.762306327 +0000 2024-03-29T10:56:31.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:56:31.958 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-29T10:56:31.958 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-29T10:56:31.958 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000156097 s, 3.3 MB/s 2024-03-29T10:56:31.959 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:56:32.015 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-03-29T10:56:32.070 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:56:32.070 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:56:32.070 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:56:32.070 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:56:32.070 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:56:32.070 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-29 10:56:23.489583248 +0000 2024-03-29T10:56:32.070 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-29 10:53:54.100299028 +0000 2024-03-29T10:56:32.070 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-29 10:53:54.100299028 +0000 2024-03-29T10:56:32.070 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-29 10:53:54.100299028 +0000 2024-03-29T10:56:32.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:56:32.135 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-29T10:56:32.136 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-29T10:56:32.136 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000187604 s, 2.7 MB/s 2024-03-29T10:56:32.137 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:56:32.192 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:32.193 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:56:32.193 INFO:tasks.ceph:Generating config... 2024-03-29T10:56:32.194 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:56:32.194 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:56:32.194 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:56:32.194 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:56:32.194 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:56:32.194 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:56:32.194 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:56:32.195 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:56:32.195 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:56:32.195 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:56:32.195 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:56:32.196 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:56:32.197 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:56:32.197 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:56:32.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:56:32.253 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:56:32.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:56:32.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:56:32.327 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.107:6789]'), ('mon.c', '[v1:172.21.15.107:6790]'), ('mon.b', '[v1:172.21.15.129:6789]')] 2024-03-29T10:56:32.328 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.107:6789],[v1:172.21.15.107:6790],[v1:172.21.15.129:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:56:32.328 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:32.329 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:56:32.383 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.107:6789]' --addv c '[v1:172.21.15.107:6790]' --addv b '[v1:172.21.15.129:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:32.468 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:32.468 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: generated fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:56:32.468 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = quincy 2024-03-29T10:56:32.468 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 0 2024-03-29T10:56:32.468 INFO:teuthology.orchestra.run.smithi107.stdout:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:56:32.468 INFO:teuthology.orchestra.run.smithi107.stdout:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:56:32.469 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:56:32.469 INFO:teuthology.orchestra.run.smithi107.stdout:min_mon_release 17 (quincy) 2024-03-29T10:56:32.469 INFO:teuthology.orchestra.run.smithi107.stdout:election_strategy: 1 2024-03-29T10:56:32.469 INFO:teuthology.orchestra.run.smithi107.stdout:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:56:32.469 INFO:teuthology.orchestra.run.smithi107.stdout:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:56:32.469 INFO:teuthology.orchestra.run.smithi107.stdout:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:56:32.469 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:56:32.470 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:56:32.485 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab... 2024-03-29T10:56:32.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:56:32.528 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:56:32.575 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2024-03-29T10:56:32.576 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2024-03-29T10:56:32.577 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2024-03-29T10:56:32.578 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "[v1:172.21.15.107:6789],[v1:172.21.15.107:6790],[v1:172.21.15.129:6789]" 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: lockdep = True 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: mon election default strategy = 1 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: ms bind msgr2 = False 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 5000 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: ms type = async 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2024-03-29T10:56:32.579 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug trim objects = true 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression algorithm = zstd 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression mode = aggressive 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore zero block detection = True 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 20 2024-03-29T10:56:32.581 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 20 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 20 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 20 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 10 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd class default list = * 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd class load list = * 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd client watch timeout = 120 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify cached snaps = True 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify missing on start = True 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:56:32.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock override recovery settings = True 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue = debug_random 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue cut off = debug_random 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: osd shutdown pgref assert = True 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.583 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:56:32.584 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon min osdmap epochs = 50 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune interval = 2 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune min = 15 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon scrub interval = 300 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on pool no app = False 2024-03-29T10:56:32.585 INFO:teuthology.orchestra.run.smithi107.stdout: paxos service trim min = 10 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 20 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: debug objecter = 20 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout: debug rados = 20 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2024-03-29T10:56:32.586 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2024-03-29T10:56:32.587 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2024-03-29T10:56:32.587 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-03-29T10:56:32.587 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-03-29T10:56:32.587 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:56:32.587 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-03-29T10:56:32.587 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.587 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-03-29T10:56:32.587 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.587 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-03-29T10:56:32.588 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-03-29T10:56:32.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "[v1:172.21.15.107:6789],[v1:172.21.15.107:6790],[v1:172.21.15.129:6789]" 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: lockdep = True 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: mon client directed command retry = 5 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: mon election default strategy = 1 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: ms bind msgr2 = False 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: ms inject socket failures = 5000 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: ms type = async 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:56:32.590 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-03-29T10:56:32.591 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-03-29T10:56:32.592 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore compression algorithm = zstd 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore compression mode = aggressive 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore zero block detection = True 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 20 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 20 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: debug monc = 20 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 10 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:56:32.593 INFO:teuthology.orchestra.run.smithi129.stdout: osd class default list = * 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd class load list = * 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd client watch timeout = 120 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify cached snaps = True 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify missing on start = True 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock override recovery settings = True 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd op queue = debug_random 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd op queue cut off = debug_random 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: osd shutdown pgref assert = True 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-03-29T10:56:32.594 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-03-29T10:56:32.595 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:56:32.596 INFO:teuthology.orchestra.run.smithi129.stdout: mon min osdmap epochs = 50 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune interval = 2 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune min = 15 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: mon scrub interval = 300 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on pool no app = False 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: paxos service trim min = 10 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: debug monc = 20 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-03-29T10:56:32.597 INFO:teuthology.orchestra.run.smithi129.stdout: debug objecter = 20 2024-03-29T10:56:32.598 INFO:teuthology.orchestra.run.smithi129.stdout: debug rados = 20 2024-03-29T10:56:32.598 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-03-29T10:56:32.598 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-03-29T10:56:32.598 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-03-29T10:56:32.599 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:56:32.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:56:32.698 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:56:32.698 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:32.698 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:56:32.713 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:32.713 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:56:32.768 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T10:56:32.768 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:32.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:56:32.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:56:32.847 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:56:32.847 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:32.903 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T10:56:32.903 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:56:32.903 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:56:32.903 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:56:32.943 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:56:32.943 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:32.998 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:56:32.998 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:56:32.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:56:33.054 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:56:33.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:56:33.118 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:56:33.121 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:56:33.121 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:56:33.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:56:33.163 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:56:33.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:56:33.220 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:56:33.236 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:56:33.236 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:56:33.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:56:33.263 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:33.263 INFO:tasks.ceph:role: osd.0 2024-03-29T10:56:33.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T10:56:33.263 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:56:33.337 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:56:33.337 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:56:33.338 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:56:33.338 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:56:33.338 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:56:33.338 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:56:33.338 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:56:33.338 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:56:33.338 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:56:33.338 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:56:33.856 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-03-29T10:56:33.858 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-03-29T10:56:33.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:56:33.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:56:34.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:56:34.078 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:34.078 INFO:tasks.ceph:role: osd.1 2024-03-29T10:56:34.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T10:56:34.078 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:56:34.153 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:56:34.153 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:56:34.153 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:56:34.153 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:56:34.154 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:56:34.154 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:56:34.154 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:56:34.154 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:56:34.154 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:56:34.154 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:56:34.616 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-03-29T10:56:34.618 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-03-29T10:56:34.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:56:34.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:56:34.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:56:34.802 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:34.802 INFO:tasks.ceph:role: osd.2 2024-03-29T10:56:34.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T10:56:34.803 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:56:34.877 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:56:35.375 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-03-29T10:56:35.377 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-03-29T10:56:35.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:56:35.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:56:35.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:56:35.639 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:35.639 INFO:tasks.ceph:role: osd.3 2024-03-29T10:56:35.639 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T10:56:35.639 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:56:35.714 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:56:35.714 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:56:35.714 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:56:35.714 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:56:35.715 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:56:35.715 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:56:35.715 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:56:35.715 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:56:35.715 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:56:35.715 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:56:36.206 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-03-29T10:56:36.207 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-03-29T10:56:36.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:56:36.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:56:36.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:36.482+0000 7fb86940c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:56:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:36.482+0000 7fb86940c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:56:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:36.482+0000 7fb86940c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:56:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:36.482+0000 7fb86940c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:56:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:36.482+0000 7fb86940c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:56:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:36.482+0000 7fb86940c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:56:39.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:56:39.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:39.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:39.470+0000 7f822f015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:56:39.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:39.470+0000 7f822f015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:56:39.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:39.470+0000 7f822f015640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:56:39.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:39.471+0000 7f822f015640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:56:39.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:39.471+0000 7f822f015640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:56:39.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:39.471+0000 7f822f015640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:56:42.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:56:42.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:42.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:42.439+0000 7fe07d024640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:56:42.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:42.439+0000 7fe07d024640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:56:42.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:42.439+0000 7fe07d024640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:56:42.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:42.439+0000 7fe07d024640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:56:42.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:42.439+0000 7fe07d024640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:56:42.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:42.440+0000 7fe07d024640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:56:45.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:56:45.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:45.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:45.397+0000 7f2b8a811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:56:45.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:45.397+0000 7f2b8a811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:56:45.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:45.397+0000 7f2b8a811640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:56:45.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:45.397+0000 7f2b8a811640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:56:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:45.398+0000 7f2b8a811640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:56:45.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:56:45.398+0000 7f2b8a811640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:56:48.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:56:48.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:56:48.291 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:48.291 INFO:tasks.ceph:role: osd.4 2024-03-29T10:56:48.292 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T10:56:48.292 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:56:48.369 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:56:48.370 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:56:48.370 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:56:48.370 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:56:48.370 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:56:48.370 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:56:48.370 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:56:48.370 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:56:48.370 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:56:48.370 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:56:48.877 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-03-29T10:56:48.879 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-03-29T10:56:48.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:56:49.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:56:49.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:56:49.136 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:49.136 INFO:tasks.ceph:role: osd.5 2024-03-29T10:56:49.136 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T10:56:49.136 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:56:49.211 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:56:49.212 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:56:49.212 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:56:49.212 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:56:49.212 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:56:49.212 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:56:49.212 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:56:49.212 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:56:49.212 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:56:49.212 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:56:49.655 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-03-29T10:56:49.657 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-03-29T10:56:49.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:56:49.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:56:49.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:56:49.866 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:49.866 INFO:tasks.ceph:role: osd.6 2024-03-29T10:56:49.866 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T10:56:49.867 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:56:49.944 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:56:49.944 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:56:49.944 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:56:49.944 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:56:49.944 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:56:49.944 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:56:49.944 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:56:49.944 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:56:49.944 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:56:49.945 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:56:50.441 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-03-29T10:56:50.444 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-03-29T10:56:50.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:56:50.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:56:50.619 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:56:50.686 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:56:50.687 INFO:tasks.ceph:role: osd.7 2024-03-29T10:56:50.689 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T10:56:50.689 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:56:50.760 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:56:50.760 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:56:50.760 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:56:50.760 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:56:50.760 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:56:50.761 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:56:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:56:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:56:50.761 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:56:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:56:51.281 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-03-29T10:56:51.283 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-03-29T10:56:51.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:56:51.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:56:51.425 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:51.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:51.547+0000 7fd95180b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:56:51.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:51.548+0000 7fd95180b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:56:51.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:51.548+0000 7fd95180b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:56:51.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:51.548+0000 7fd95180b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:56:51.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:51.548+0000 7fd95180b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:56:51.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:51.548+0000 7fd95180b640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:56:54.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:56:54.405 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:54.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:54.511+0000 7f2b60c8b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:56:54.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:54.511+0000 7f2b60c8b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:56:54.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:54.512+0000 7f2b60c8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:56:54.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:54.512+0000 7f2b60c8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:56:54.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:54.512+0000 7f2b60c8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:56:54.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:54.512+0000 7f2b60c8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:56:57.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:56:57.367 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:56:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:57.475+0000 7faa3040d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:56:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:57.475+0000 7faa3040d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:56:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:57.475+0000 7faa3040d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:56:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:57.475+0000 7faa3040d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:56:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:57.475+0000 7faa3040d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:56:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:56:57.476+0000 7faa3040d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:57:00.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:57:00.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:57:00.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:00.446+0000 7fda7e211640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:57:00.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:00.446+0000 7fda7e211640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:57:00.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:00.446+0000 7fda7e211640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:57:00.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:00.446+0000 7fda7e211640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:57:00.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:00.446+0000 7fda7e211640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:57:00.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:00.447+0000 7fda7e211640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:57:03.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:57:03.306 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:57:03.306 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:03.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:57:03.339 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:03.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:57:03.409 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:03.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:57:03.478 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:03.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:57:03.547 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:03.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:57:03.616 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:03.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:57:03.647 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:03.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:57:03.718 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:03.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:57:03.789 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:03.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:57:03.859 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:03.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:57:03.928 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:03.929 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:57:03.943 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:03.943 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:57:03.985 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:57:03.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:57:03.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:57:04.028 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.y] 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBhngZmlVhxBBAAbr4r5sevBaefkQC/x73NCQ== 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBkngZm19fJHBAAH3G4074m7CNhhEmw2U+kVw== 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBnngZmUmMcHBAAmEdd+XZRwjKqfSTVoEwznQ== 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBqngZmnDJAGhAApV+7aL79yZ+UaUmvp/nxmg== 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBtngZm227CFxAAz63xJG1h93GpS5G+ibbkGA== 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBhngZmCTBLCBAAmjWB7VgnBOGVWlPMiUBqwQ== 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.4] 2024-03-29T10:57:04.029 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBzngZmwj7DIBAAh+e6QgcTbvGyc8PTo7p0NA== 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.5] 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQB2ngZmQdScHhAARxIAMFeipBGtkZUqxEU9QA== 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.6] 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQB5ngZmHjByHBAA6zbfIFctVjI7qV/2BT88JA== 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.7] 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQB8ngZmlCypGhAApRTsacgcGMoj577k+/0pjQ== 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBhngZmSiT1ChAAoJUU7/smFpmIFaJv5ImvqA== 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout:[client.1] 2024-03-29T10:57:04.030 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBhngZm1Q5iDhAAD70Z5PnhJpO7KAl9WH/9sQ== 2024-03-29T10:57:04.053 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.y] 2024-03-29T10:57:04.053 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBhngZmlVhxBBAAbr4r5sevBaefkQC/x73NCQ== 2024-03-29T10:57:04.053 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-03-29T10:57:04.053 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBkngZm19fJHBAAH3G4074m7CNhhEmw2U+kVw== 2024-03-29T10:57:04.053 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-03-29T10:57:04.053 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBnngZmUmMcHBAAmEdd+XZRwjKqfSTVoEwznQ== 2024-03-29T10:57:04.053 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-03-29T10:57:04.053 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBqngZmnDJAGhAApV+7aL79yZ+UaUmvp/nxmg== 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.3] 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBtngZm227CFxAAz63xJG1h93GpS5G+ibbkGA== 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBhngZmCTBLCBAAmjWB7VgnBOGVWlPMiUBqwQ== 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.4] 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBzngZmwj7DIBAAh+e6QgcTbvGyc8PTo7p0NA== 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.5] 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB2ngZmQdScHhAARxIAMFeipBGtkZUqxEU9QA== 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.6] 2024-03-29T10:57:04.054 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB5ngZmHjByHBAA6zbfIFctVjI7qV/2BT88JA== 2024-03-29T10:57:04.055 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.7] 2024-03-29T10:57:04.055 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQB8ngZmlCypGhAApRTsacgcGMoj577k+/0pjQ== 2024-03-29T10:57:04.055 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-03-29T10:57:04.055 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBhngZmSiT1ChAAoJUU7/smFpmIFaJv5ImvqA== 2024-03-29T10:57:04.055 INFO:teuthology.orchestra.run.smithi129.stdout:[client.1] 2024-03-29T10:57:04.055 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBhngZm1Q5iDhAAD70Z5PnhJpO7KAl9WH/9sQ== 2024-03-29T10:57:04.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:57:04.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:57:04.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.153 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.261 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:57:04.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:57:04.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.475 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.529 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:57:04.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:57:04.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:57:04.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:57:04.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:57:04.748 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:57:04.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:57:04.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:57:04.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:57:04.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:57:04.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:57:05.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:57:05.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:57:05.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:57:05.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:57:05.242 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:57:05.245 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:57:05.297 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:57:05.297 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:57:05.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:57:05.299 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:57:05.300 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:57:05.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:57:05.301 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:57:05.302 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:57:05.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:57:05.340 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:57:05.340 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:57:05.340 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:57:05.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:57:05.342 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:57:05.342 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:57:05.342 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:57:05.344 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:57:05.345 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:57:05.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:57:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.465+0000 7f3b291c6640 10 monclient: get_monmap_and_config 2024-03-29T10:57:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.465+0000 7f3b291c6640 10 monclient: build_initial_monmap 2024-03-29T10:57:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.465+0000 7f3b291c6640 10 monclient: monmap: 2024-03-29T10:57:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2024-03-29T10:57:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:57:05.466210+0000 2024-03-29T10:57:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:57:05.466210+0000 2024-03-29T10:57:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:05.447 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.465+0000 7f3b291c6640 1 Processor -- start 2024-03-29T10:57:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.465+0000 7f3b291c6640 1 -- start start 2024-03-29T10:57:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.465+0000 7f3b291c6640 10 monclient: init 2024-03-29T10:57:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3b2409ac40 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3b2409c870 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3b240b4360 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2409dec0 con 0x7f3b2409ac40 2024-03-29T10:57:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2409da30 con 0x7f3b240b4360 2024-03-29T10:57:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240b39c0 con 0x7f3b2409c870 2024-03-29T10:57:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b22d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3b2409ac40 0x7f3b240b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56458/0 (socket says 172.21.15.107:56458) 2024-03-29T10:57:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b22d9d640 1 -- 172.21.15.107:0/3516253300 learned_addr learned my addr 172.21.15.107:0/3516253300 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:05.466+0000 7f3b291c6640 10 monclient(hunting): authenticate will time out at 756.734617s 2024-03-29T10:57:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 10 monclient(hunting): tick 2024-03-29T10:57:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:57:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 1 monclient(hunting): continuing hunt 2024-03-29T10:57:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 1 -- 172.21.15.107:0/3516253300 >> v1:172.21.15.107:6790/0 conn(0x7f3b2409c870 legacy=0x7f3b2413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 1 -- 172.21.15.107:0/3516253300 >> v1:172.21.15.129:6789/0 conn(0x7f3b240b4360 legacy=0x7f3b2414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 1 -- 172.21.15.107:0/3516253300 >> v1:172.21.15.107:6789/0 conn(0x7f3b2409ac40 legacy=0x7f3b240b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 10 monclient(hunting): picked mon.noname-b con 0x7f3b100036e0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 10 monclient(hunting): picked mon.noname-a con 0x7f3b100072d0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 10 monclient(hunting): picked mon.noname-c con 0x7f3b1000afd0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 1 -- 172.21.15.107:0/3516253300 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240b5160 con 0x7f3b100072d0 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 1 -- 172.21.15.107:0/3516253300 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1000fca0 con 0x7f3b1000afd0 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 1 -- 172.21.15.107:0/3516253300 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b10010ee0 con 0x7f3b100036e0 2024-03-29T10:57:08.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:08.466+0000 7f3b20d99640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.636+0000 7f3b21d9b640 1 -- 172.21.15.107:0/3516253300 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488865182 0 0) 0x7f3b240b5160 con 0x7f3b100072d0 2024-03-29T10:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.636+0000 7f3b21d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.636+0000 7f3b21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.636+0000 7f3b21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.636+0000 7f3b21d9b640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.636+0000 7f3b21d9b640 1 -- 172.21.15.107:0/3516253300 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b0c003520 con 0x7f3b100072d0 2024-03-29T10:57:10.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 1 -- 172.21.15.107:0/3516253300 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005989641 0 0) 0x7f3b0c003520 con 0x7f3b100072d0 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 1 -- 172.21.15.107:0/3516253300 >> v1:172.21.15.107:6790/0 conn(0x7f3b100036e0 legacy=0x7f3b10005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 1 -- 172.21.15.107:0/3516253300 >> v1:172.21.15.129:6789/0 conn(0x7f3b1000afd0 legacy=0x7f3b1000d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 1 monclient: found mon.noname-a 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 1 -- 172.21.15.107:0/3516253300 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b10012140 con 0x7f3b100072d0 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 10 monclient: _finish_auth 0 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 1 -- 172.21.15.107:0/3516253300 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3b18001600 con 0x7f3b100072d0 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.638+0000 7f3b21d9b640 10 monclient: dump: 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b291c6640 5 monclient: authenticate success, global_id 4101 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b291c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b21d9b640 1 -- 172.21.15.107:0/3516253300 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b180017d0 con 0x7f3b100072d0 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b21d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:10.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b21d9b640 1 -- 172.21.15.107:0/3516253300 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3b18001b60 con 0x7f3b100072d0 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b21d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b21d9b640 10 monclient: dump: 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b291c6640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:10.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.639+0000 7f3b291c6640 4 set_mon_vals no callback set 2024-03-29T10:57:10.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.640+0000 7f3b291c6640 10 monclient: shutdown 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.640+0000 7f3b291c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.640+0000 7f3b291c6640 1 -- 172.21.15.107:0/3516253300 >> v1:172.21.15.107:6789/0 conn(0x7f3b100072d0 legacy=0x7f3b10009740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.640+0000 7f3b291c6640 1 -- 172.21.15.107:0/3516253300 shutdown_connections 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.640+0000 7f3b291c6640 1 -- 172.21.15.107:0/3516253300 >> 172.21.15.107:0/3516253300 conn(0x7f3b24066b00 msgr2=0x7f3b24066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.640+0000 7f3b291c6640 1 -- 172.21.15.107:0/3516253300 shutdown_connections 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.640+0000 7f3b291c6640 1 -- 172.21.15.107:0/3516253300 wait complete. 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.641+0000 7f3b291c6640 10 monclient: build_initial_monmap 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.641+0000 7f3b291c6640 10 monclient: monmap: 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:10.628 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.642+0000 7f3b291c6640 1 librados: starting msgr at 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.642+0000 7f3b291c6640 1 librados: starting objecter 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.642+0000 7f3b291c6640 1 Processor -- start 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.642+0000 7f3b291c6640 1 -- start start 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.642+0000 7f3b291c6640 1 librados: setting wanted keys 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.642+0000 7f3b291c6640 1 librados: calling monclient init 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.642+0000 7f3b291c6640 10 monclient: init 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3b100036e0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:10.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3b100072d0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3b1000afd0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b241417b0 con 0x7f3b100036e0 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b24142840 con 0x7f3b100072d0 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b24140d60 con 0x7f3b1000afd0 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b291c6640 10 monclient(hunting): authenticate will time out at 761.911652s 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b22d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3b100036e0 0x7f3b24140650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56510/0 (socket says 172.21.15.107:56510) 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b22d9d640 1 -- 172.21.15.107:0/187634925 learned_addr learned my addr 172.21.15.107:0/187634925 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.643+0000 7f3b0b7fe640 10 client.?.objecter ms_handle_connect 0x7f3b100036e0 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 client.?.objecter ms_handle_connect 0x7f3b100072d0 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 client.?.objecter ms_handle_connect 0x7f3b1000afd0 2024-03-29T10:57:10.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991384530 0 0) 0x7f3b241417b0 con 0x7f3b100036e0 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b10009a30 con 0x7f3b100036e0 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1605642272 0 0) 0x7f3b10009a30 con 0x7f3b100036e0 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b241417b0 con 0x7f3b100036e0 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3b14001700 con 0x7f3b100036e0 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1309776943 0 0) 0x7f3b241417b0 con 0x7f3b100036e0 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 >> v1:172.21.15.107:6790/0 conn(0x7f3b1000afd0 legacy=0x7f3b241da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 >> v1:172.21.15.129:6789/0 conn(0x7f3b100072d0 legacy=0x7f3b24135e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 monclient: found mon.a 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b24142ab0 con 0x7f3b100036e0 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b291c6640 5 monclient: authenticate success, global_id 4113 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b291c6640 10 monclient: _renew_subs 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b291c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b291c6640 1 -- 172.21.15.107:0/187634925 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b241414b0 con 0x7f3b100036e0 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b291c6640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b291c6640 10 monclient: _renew_subs 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b291c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b291c6640 1 -- 172.21.15.107:0/187634925 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b24136760 con 0x7f3b100036e0 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.644+0000 7f3b291c6640 1 librados: init done 2024-03-29T10:57:10.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b14001cb0 con 0x7f3b100036e0 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3b14004980 con 0x7f3b100036e0 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 10 monclient: dump: 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:10.634 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b2159a640 4 set_mon_vals no callback set 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3b14004e70 con 0x7f3b100036e0 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3702581394 0 0) 0x7f3b140063a0 con 0x7f3b100036e0 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 10 client.4113.objecter ms_dispatch 0x7f3b24000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.645+0000 7f3b0b7fe640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.646+0000 7f3b291c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.646+0000 7f3b291c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.646+0000 7f3b291c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.646+0000 7f3b291c6640 1 -- 172.21.15.107:0/187634925 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b04005080 con 0x7f3b100036e0 2024-03-29T10:57:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.649+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3b140053c0 con 0x7f3b100036e0 2024-03-29T10:57:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.649+0000 7f3b0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.649+0000 7f3b0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.719+0000 7f3b291c6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:57:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.719+0000 7f3b291c6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:57:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.719+0000 7f3b291c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:10.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.719+0000 7f3b291c6640 1 -- 172.21.15.107:0/187634925 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3b04005330 con 0x7f3b100036e0 2024-03-29T10:57:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.721+0000 7f3b0b7fe640 1 -- 172.21.15.107:0/187634925 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f3b140056c0 con 0x7f3b100036e0 2024-03-29T10:57:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.721+0000 7f3b0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:57:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.721+0000 7f3b0b7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:57:10.701 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 10 librados: watch_flush enter 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 10 librados: watch_flush exit 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 10 client.4113.objecter successfully canceled tick 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 10 monclient: shutdown 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 1 -- 172.21.15.107:0/187634925 >> v1:172.21.15.107:6789/0 conn(0x7f3b100036e0 legacy=0x7f3b24140650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 1 -- 172.21.15.107:0/187634925 shutdown_connections 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 1 -- 172.21.15.107:0/187634925 >> 172.21.15.107:0/187634925 conn(0x7f3b24066b00 msgr2=0x7f3b24069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:10.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 1 -- 172.21.15.107:0/187634925 shutdown_connections 2024-03-29T10:57:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 1 -- 172.21.15.107:0/187634925 wait complete. 2024-03-29T10:57:10.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:10.722+0000 7f3b291c6640 1 librados: shutdown 2024-03-29T10:57:10.713 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:57:10.713 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:57:10.714 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:10.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:57:10.740 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:10.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:57:10.806 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:10.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:57:10.874 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-29T10:57:10.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:57:10.941 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:10.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:57:10.969 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:10.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:57:11.036 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:11.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:57:11.094 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:11.113+0000 7f1b61413040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:57:11.100 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:11.119+0000 7fc49e442040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:57:11.104 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-29T10:57:11.104 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:57:11.147 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:11.167+0000 7f1b61413040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:57:11.155 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:11.175+0000 7fc49e442040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:57:11.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 9629f157-44b7-489b-9c0a-2f18a8223dc2 0 2024-03-29T10:57:11.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.330+0000 7f61a082a640 10 monclient: get_monmap_and_config 2024-03-29T10:57:11.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.330+0000 7f61a082a640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.330+0000 7f61a082a640 10 monclient: monmap: 2024-03-29T10:57:11.311 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:11.311 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.312 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:57:11.331848+0000 2024-03-29T10:57:11.312 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:57:11.331848+0000 2024-03-29T10:57:11.312 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.312 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.312 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:11.312 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:11.312 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:11.312 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.331+0000 7f61a082a640 1 Processor -- start 2024-03-29T10:57:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.331+0000 7f61a082a640 1 -- start start 2024-03-29T10:57:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.331+0000 7f61a082a640 10 monclient: init 2024-03-29T10:57:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 10 monclient(hunting): picked mon.noname-b con 0x7f619c09ac40 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 10 monclient(hunting): picked mon.noname-a con 0x7f619c09c870 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 10 monclient(hunting): picked mon.noname-c con 0x7f619c0b4360 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c09dec0 con 0x7f619c09c870 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c09da30 con 0x7f619c0b4360 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c0b39c0 con 0x7f619c09ac40 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 10 monclient(hunting): authenticate will time out at 768.644264s 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f619ad9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f619c0b4360 0x7f619c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60524/0 (socket says 172.21.15.129:60524) 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f619ad9d640 1 -- 172.21.15.129:0/3581084744 learned_addr learned my addr 172.21.15.129:0/3581084744 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162086444 0 0) 0x7f619c09da30 con 0x7f619c0b4360 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188003520 con 0x7f619c0b4360 2024-03-29T10:57:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194345379 0 0) 0x7f619c0b39c0 con 0x7f619c09ac40 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): my global_id is 4103 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c09da30 con 0x7f619c09ac40 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015406820 0 0) 0x7f6184001040 con 0x7f619c09c870 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c0b39c0 con 0x7f619c09c870 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216381002 0 0) 0x7f6188003520 con 0x7f619c0b4360 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 >> v1:172.21.15.107:6790/0 conn(0x7f619c09ac40 legacy=0x7f619c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 >> v1:172.21.15.107:6789/0 conn(0x7f619c09c870 legacy=0x7f619c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 monclient: found mon.noname-c 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c0b5160 con 0x7f619c0b4360 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6194001a10 con 0x7f619c0b4360 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 5 monclient: authenticate success, global_id 4108 2024-03-29T10:57:11.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f6199d9b640 10 monclient: dump: 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.332+0000 7f61a082a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:11.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6194001b70 con 0x7f619c0b4360 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f6199d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f6199d9b640 1 -- 172.21.15.129:0/3581084744 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6194001f80 con 0x7f619c0b4360 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f61a082a640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f61a082a640 4 set_mon_vals no callback set 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f6199d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f6199d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f6199d9b640 10 monclient: dump: 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.325 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f61a082a640 10 monclient: shutdown 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f61a082a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f61a082a640 1 -- 172.21.15.129:0/3581084744 >> v1:172.21.15.129:6789/0 conn(0x7f619c0b4360 legacy=0x7f619c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f61a082a640 1 -- 172.21.15.129:0/3581084744 shutdown_connections 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f61a082a640 1 -- 172.21.15.129:0/3581084744 >> 172.21.15.129:0/3581084744 conn(0x7f619c066b00 msgr2=0x7f619c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f61a082a640 1 -- 172.21.15.129:0/3581084744 shutdown_connections 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.333+0000 7f61a082a640 1 -- 172.21.15.129:0/3581084744 wait complete. 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.334+0000 7f61a082a640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.334+0000 7f61a082a640 10 monclient: monmap: 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.334+0000 7f61a082a640 1 librados: starting msgr at 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.334+0000 7f61a082a640 1 librados: starting objecter 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.334+0000 7f61a082a640 1 Processor -- start 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.334+0000 7f61a082a640 1 -- start start 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.334+0000 7f61a082a640 1 librados: setting wanted keys 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.334+0000 7f61a082a640 1 librados: calling monclient init 2024-03-29T10:57:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.334+0000 7f61a082a640 10 monclient: init 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 10 monclient(hunting): picked mon.noname-c con 0x7f619c09ac40 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 10 monclient(hunting): picked mon.noname-a con 0x7f619c1d8690 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 10 monclient(hunting): picked mon.noname-b con 0x7f619c1455c0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c1d7af0 con 0x7f619c1d8690 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c145a10 con 0x7f619c1455c0 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c0b5a20 con 0x7f619c09ac40 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f61a082a640 10 monclient(hunting): authenticate will time out at 768.647297s 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f619ad9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f619c1455c0 0x7f619c1e2400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60536/0 (socket says 172.21.15.129:60536) 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f619ad9d640 1 -- 172.21.15.129:0/17688138 learned_addr learned my addr 172.21.15.129:0/17688138 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 client.?.objecter ms_handle_connect 0x7f619c1455c0 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 client.?.objecter ms_handle_connect 0x7f619c1d8690 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 client.?.objecter ms_handle_connect 0x7f619c09ac40 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681351486 0 0) 0x7f619c145a10 con 0x7f619c1455c0 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617c004eb0 con 0x7f619c1455c0 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512570406 0 0) 0x7f619c0b5a20 con 0x7f619c09ac40 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c145a10 con 0x7f619c09ac40 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872555695 0 0) 0x7f619c1d7af0 con 0x7f619c1d8690 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c0b5a20 con 0x7f619c1d8690 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3591657397 0 0) 0x7f617c004eb0 con 0x7f619c1455c0 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f619c1d7af0 con 0x7f619c1455c0 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6194001ec0 con 0x7f619c1455c0 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1976664494 0 0) 0x7f619c145a10 con 0x7f619c09ac40 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.335+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f617c004eb0 con 0x7f619c09ac40 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f618c001970 con 0x7f619c09ac40 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3124912719 0 0) 0x7f619c1d7af0 con 0x7f619c1455c0 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 >> v1:172.21.15.107:6790/0 conn(0x7f619c09ac40 legacy=0x7f619c1d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 >> v1:172.21.15.107:6789/0 conn(0x7f619c1d8690 legacy=0x7f619c144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 monclient: found mon.b 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c141960 con 0x7f619c1455c0 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f61a082a640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f61a082a640 10 monclient: _renew_subs 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f61a082a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f61a082a640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c1e3b20 con 0x7f619c1455c0 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f61a082a640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f61a082a640 10 monclient: _renew_subs 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f61a082a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f61a082a640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c1e3e90 con 0x7f619c1455c0 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f61a082a640 1 librados: init done 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6194004e10 con 0x7f619c1455c0 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6194005120 con 0x7f619c1455c0 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f619959a640 4 set_mon_vals no callback set 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 monclient: dump: 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6194005610 con 0x7f619c1455c0 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3172249348 0 0) 0x7f6194006950 con 0x7f619c1455c0 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 10 client.4111.objecter ms_dispatch 0x7f619c000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.336+0000 7f6192ffd640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.337+0000 7f61a082a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.337+0000 7f61a082a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.337+0000 7f61a082a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.337+0000 7f61a082a640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6180005080 con 0x7f619c1455c0 2024-03-29T10:57:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.339+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6194006b10 con 0x7f619c1455c0 2024-03-29T10:57:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.339+0000 7f6192ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.339+0000 7f6192ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:11.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.409+0000 7f61a082a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9629f157-44b7-489b-9c0a-2f18a8223dc2", "id": 0}] 2024-03-29T10:57:11.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.409+0000 7f61a082a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9629f157-44b7-489b-9c0a-2f18a8223dc2", "id": 0}] 2024-03-29T10:57:11.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.409+0000 7f61a082a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.409+0000 7f61a082a640 1 -- 172.21.15.129:0/17688138 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9629f157-44b7-489b-9c0a-2f18a8223dc2", "id": 0} v 0) v1 -- 0x7f61800058b0 con 0x7f619c1455c0 2024-03-29T10:57:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.415+0000 7f6192ffd640 1 -- 172.21.15.129:0/17688138 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9629f157-44b7-489b-9c0a-2f18a8223dc2", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 309607021 0 732707515) 0x7f6194005a90 con 0x7f619c1455c0 2024-03-29T10:57:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.415+0000 7f6192ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9629f157-44b7-489b-9c0a-2f18a8223dc2", "id": 0}] 2024-03-29T10:57:11.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.415+0000 7f6192ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:11.395 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-03-29T10:57:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.415+0000 7f61a082a640 10 librados: watch_flush enter 2024-03-29T10:57:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.415+0000 7f61a082a640 10 librados: watch_flush exit 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.415+0000 7f61a082a640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.415+0000 7f61a082a640 10 client.4111.objecter successfully canceled tick 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.416+0000 7f61a082a640 10 monclient: shutdown 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.416+0000 7f61a082a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.416+0000 7f61a082a640 1 -- 172.21.15.129:0/17688138 >> v1:172.21.15.129:6789/0 conn(0x7f619c1455c0 legacy=0x7f619c1e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.416+0000 7f61a082a640 1 -- 172.21.15.129:0/17688138 shutdown_connections 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.416+0000 7f61a082a640 1 -- 172.21.15.129:0/17688138 >> 172.21.15.129:0/17688138 conn(0x7f619c066b00 msgr2=0x7f619c13a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.416+0000 7f61a082a640 1 -- 172.21.15.129:0/17688138 shutdown_connections 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.416+0000 7f61a082a640 1 -- 172.21.15.129:0/17688138 wait complete. 2024-03-29T10:57:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.416+0000 7f61a082a640 1 librados: shutdown 2024-03-29T10:57:11.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new f67bee6c-2441-47fe-87e6-7067a828fad9 1 2024-03-29T10:57:11.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.526+0000 7f2fdb989640 10 monclient: get_monmap_and_config 2024-03-29T10:57:11.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.526+0000 7f2fdb989640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.526+0000 7f2fdb989640 10 monclient: monmap: 2024-03-29T10:57:11.507 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:11.507 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.507 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:57:11.527824+0000 2024-03-29T10:57:11.508 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:57:11.527824+0000 2024-03-29T10:57:11.508 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.508 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.508 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:11.508 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:11.508 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:11.508 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.527+0000 7f2fdb989640 1 Processor -- start 2024-03-29T10:57:11.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.527+0000 7f2fdb989640 1 -- start start 2024-03-29T10:57:11.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.527+0000 7f2fdb989640 10 monclient: init 2024-03-29T10:57:11.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:11.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 10 monclient(hunting): picked mon.noname-b con 0x7f2fd409ac40 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd409c870 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd40b4360 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd409dec0 con 0x7f2fd409c870 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd409da30 con 0x7f2fd40b4360 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd40b39c0 con 0x7f2fd409ac40 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fdb989640 10 monclient(hunting): authenticate will time out at 768.840302s 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fd9f26640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f2fd40b4360 0x7f2fd414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60544/0 (socket says 172.21.15.129:60544) 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fd9f26640 1 -- 172.21.15.129:0/343411321 learned_addr learned my addr 172.21.15.129:0/343411321 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989175922 0 0) 0x7f2fd409da30 con 0x7f2fd40b4360 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:57:11.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc0003520 con 0x7f2fd40b4360 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143537908 0 0) 0x7f2fd40b39c0 con 0x7f2fd409ac40 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd409da30 con 0x7f2fd409ac40 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050279722 0 0) 0x7f2fc4001110 con 0x7f2fd409c870 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd40b39c0 con 0x7f2fd409c870 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 228624077 0 0) 0x7f2fc0003520 con 0x7f2fd40b4360 2024-03-29T10:57:11.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.528+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 >> v1:172.21.15.107:6790/0 conn(0x7f2fd409ac40 legacy=0x7f2fd40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 >> v1:172.21.15.107:6789/0 conn(0x7f2fd409c870 legacy=0x7f2fd413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 1 monclient: found mon.noname-c 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40b5160 con 0x7f2fd40b4360 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2fcc001a10 con 0x7f2fd40b4360 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:11.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: dump: 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fcc001bf0 con 0x7f2fd40b4360 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 1 -- 172.21.15.129:0/343411321 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2fcc002000 con 0x7f2fd40b4360 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fcbfff640 10 monclient: dump: 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 4 set_mon_vals no callback set 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 10 monclient: shutdown 2024-03-29T10:57:11.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 1 -- 172.21.15.129:0/343411321 >> v1:172.21.15.129:6789/0 conn(0x7f2fd40b4360 legacy=0x7f2fd414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 1 -- 172.21.15.129:0/343411321 shutdown_connections 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 1 -- 172.21.15.129:0/343411321 >> 172.21.15.129:0/343411321 conn(0x7f2fd4066b00 msgr2=0x7f2fd4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 1 -- 172.21.15.129:0/343411321 shutdown_connections 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.529+0000 7f2fdb989640 1 -- 172.21.15.129:0/343411321 wait complete. 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.530+0000 7f2fdb989640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.530+0000 7f2fdb989640 10 monclient: monmap: 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.522 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.530+0000 7f2fdb989640 1 librados: starting msgr at 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.530+0000 7f2fdb989640 1 librados: starting objecter 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.530+0000 7f2fdb989640 1 Processor -- start 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.530+0000 7f2fdb989640 1 -- start start 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.530+0000 7f2fdb989640 1 librados: setting wanted keys 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.530+0000 7f2fdb989640 1 librados: calling monclient init 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.530+0000 7f2fdb989640 10 monclient: init 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd409ac40 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 10 monclient(hunting): picked mon.noname-b con 0x7f2fd4144120 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:11.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd41d7ac0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd4146240 con 0x7f2fd409ac40 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd40b6af0 con 0x7f2fd4144120 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd41464b0 con 0x7f2fd41d7ac0 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 10 monclient(hunting): authenticate will time out at 768.843153s 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fd8f24640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f2fd4144120 0x7f2fd41d73b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60550/0 (socket says 172.21.15.129:60550) 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fd8f24640 1 -- 172.21.15.129:0/1516264056 learned_addr learned my addr 172.21.15.129:0/1516264056 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 client.?.objecter ms_handle_connect 0x7f2fd4144120 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 client.?.objecter ms_handle_connect 0x7f2fd409ac40 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 client.?.objecter ms_handle_connect 0x7f2fd41d7ac0 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385423265 0 0) 0x7f2fd40b6af0 con 0x7f2fd4144120 2024-03-29T10:57:11.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb8004eb0 con 0x7f2fd4144120 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242846647 0 0) 0x7f2fd4146240 con 0x7f2fd409ac40 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd40b6af0 con 0x7f2fd409ac40 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943364533 0 0) 0x7f2fd41464b0 con 0x7f2fd41d7ac0 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd4146240 con 0x7f2fd41d7ac0 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3649494317 0 0) 0x7f2fb8004eb0 con 0x7f2fd4144120 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fd41464b0 con 0x7f2fd4144120 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2fc4001860 con 0x7f2fd4144120 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): dump: 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.526 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1325433293 0 0) 0x7f2fd41464b0 con 0x7f2fd4144120 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 >> v1:172.21.15.107:6790/0 conn(0x7f2fd41d7ac0 legacy=0x7f2fd41e1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 >> v1:172.21.15.107:6789/0 conn(0x7f2fd409ac40 legacy=0x7f2fd4144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 monclient: found mon.b 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd41e62b0 con 0x7f2fd4144120 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 10 monclient: _renew_subs 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fdb989640 1 -- 172.21.15.129:0/1516264056 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd41e3280 con 0x7f2fd4144120 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fc4003610 con 0x7f2fd4144120 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2fc4003a60 con 0x7f2fd4144120 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fc9ffb640 10 monclient: dump: 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.528 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.531+0000 7f2fcb7fe640 4 set_mon_vals no callback set 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fdb989640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fdb989640 10 monclient: _renew_subs 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fdb989640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fdb989640 1 -- 172.21.15.129:0/1516264056 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd41e3800 con 0x7f2fd4144120 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2fc4003fd0 con 0x7f2fd4144120 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fdb989640 1 librados: init done 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2188954385 0 0) 0x7f2fc4005b70 con 0x7f2fd4144120 2024-03-29T10:57:11.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fc9ffb640 10 client.4117.objecter ms_dispatch 0x7f2fd4000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fc9ffb640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fc9ffb640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fc9ffb640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fdb989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fdb989640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fdb989640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.532+0000 7f2fdb989640 1 -- 172.21.15.129:0/1516264056 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd4074bb0 con 0x7f2fd4144120 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.534+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2fd4074bb0 con 0x7f2fd4144120 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.534+0000 7f2fc9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.534+0000 7f2fc9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.605+0000 7f2fdb989640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f67bee6c-2441-47fe-87e6-7067a828fad9", "id": 1}] 2024-03-29T10:57:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.605+0000 7f2fdb989640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f67bee6c-2441-47fe-87e6-7067a828fad9", "id": 1}] 2024-03-29T10:57:11.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.605+0000 7f2fdb989640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.605+0000 7f2fdb989640 1 -- 172.21.15.129:0/1516264056 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f67bee6c-2441-47fe-87e6-7067a828fad9", "id": 1} v 0) v1 -- 0x7f2fd409b020 con 0x7f2fd4144120 2024-03-29T10:57:11.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.611+0000 7f2fc9ffb640 1 -- 172.21.15.129:0/1516264056 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f67bee6c-2441-47fe-87e6-7067a828fad9", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1225269807 0 940483276) 0x7f2fc4004280 con 0x7f2fd4144120 2024-03-29T10:57:11.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.611+0000 7f2fc9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f67bee6c-2441-47fe-87e6-7067a828fad9", "id": 1}] 2024-03-29T10:57:11.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.611+0000 7f2fc9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:11.592 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-03-29T10:57:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 10 librados: watch_flush enter 2024-03-29T10:57:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 10 librados: watch_flush exit 2024-03-29T10:57:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:57:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:57:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 10 monclient: shutdown 2024-03-29T10:57:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 1 -- 172.21.15.129:0/1516264056 >> v1:172.21.15.129:6789/0 conn(0x7f2fd4144120 legacy=0x7f2fd41d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 1 -- 172.21.15.129:0/1516264056 shutdown_connections 2024-03-29T10:57:11.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 1 -- 172.21.15.129:0/1516264056 >> 172.21.15.129:0/1516264056 conn(0x7f2fd4066b00 msgr2=0x7f2fd413a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.612+0000 7f2fdb989640 1 -- 172.21.15.129:0/1516264056 shutdown_connections 2024-03-29T10:57:11.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.613+0000 7f2fdb989640 1 -- 172.21.15.129:0/1516264056 wait complete. 2024-03-29T10:57:11.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.613+0000 7f2fdb989640 1 librados: shutdown 2024-03-29T10:57:11.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new e75d2a72-7727-4449-bcc3-d14b92fa0ce8 2 2024-03-29T10:57:11.667 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:11.687+0000 7f1b61413040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.723+0000 7fedaac93640 10 monclient: get_monmap_and_config 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.723+0000 7fedaac93640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.723+0000 7fedaac93640 10 monclient: monmap: 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:57:11.724427+0000 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:57:11.724427+0000 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:11.705 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 1 Processor -- start 2024-03-29T10:57:11.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 1 -- start start 2024-03-29T10:57:11.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 10 monclient: init 2024-03-29T10:57:11.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:11.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 10 monclient(hunting): picked mon.noname-b con 0x7feda409ac40 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 10 monclient(hunting): picked mon.noname-a con 0x7feda409c870 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 10 monclient(hunting): picked mon.noname-c con 0x7feda40b4360 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda409dec0 con 0x7feda409c870 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda409da30 con 0x7feda40b4360 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40b39c0 con 0x7feda409ac40 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.724+0000 7fedaac93640 10 monclient(hunting): authenticate will time out at 769.036834s 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda9230640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7feda40b4360 0x7feda414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60562/0 (socket says 172.21.15.129:60562) 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda9230640 1 -- 172.21.15.129:0/4215039739 learned_addr learned my addr 172.21.15.129:0/4215039739 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287537681 0 0) 0x7feda409da30 con 0x7feda40b4360 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed94003520 con 0x7feda40b4360 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282529238 0 0) 0x7feda40b39c0 con 0x7feda409ac40 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda409da30 con 0x7feda409ac40 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066834674 0 0) 0x7fed90001110 con 0x7feda409c870 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda40b39c0 con 0x7feda409c870 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2497249890 0 0) 0x7fed94003520 con 0x7feda40b4360 2024-03-29T10:57:11.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 >> v1:172.21.15.107:6790/0 conn(0x7feda409ac40 legacy=0x7feda40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 >> v1:172.21.15.107:6789/0 conn(0x7feda409c870 legacy=0x7feda413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 monclient: found mon.noname-c 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda40b5160 con 0x7feda40b4360 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fed9c001a10 con 0x7feda40b4360 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7fedaac93640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7fedaac93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:11.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: dump: 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7fedaac93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed9c001bf0 con 0x7feda40b4360 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7fedaac93640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 1 -- 172.21.15.129:0/4215039739 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fed9c002000 con 0x7feda40b4360 2024-03-29T10:57:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7fedaac93640 4 set_mon_vals no callback set 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.725+0000 7feda37fe640 10 monclient: dump: 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.726+0000 7fedaac93640 10 monclient: shutdown 2024-03-29T10:57:11.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.726+0000 7fedaac93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.726+0000 7fedaac93640 1 -- 172.21.15.129:0/4215039739 >> v1:172.21.15.129:6789/0 conn(0x7feda40b4360 legacy=0x7feda414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.726+0000 7fedaac93640 1 -- 172.21.15.129:0/4215039739 shutdown_connections 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.726+0000 7fedaac93640 1 -- 172.21.15.129:0/4215039739 >> 172.21.15.129:0/4215039739 conn(0x7feda4066b00 msgr2=0x7feda4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.726+0000 7fedaac93640 1 -- 172.21.15.129:0/4215039739 shutdown_connections 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.726+0000 7fedaac93640 1 -- 172.21.15.129:0/4215039739 wait complete. 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.726+0000 7fedaac93640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.726+0000 7fedaac93640 10 monclient: monmap: 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.719 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 1 librados: starting msgr at 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 1 librados: starting objecter 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 1 Processor -- start 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 1 -- start start 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 1 librados: setting wanted keys 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 1 librados: calling monclient init 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 10 monclient: init 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 10 monclient(hunting): picked mon.noname-c con 0x7feda409ac40 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 10 monclient(hunting): picked mon.noname-b con 0x7feda41d8690 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 10 monclient(hunting): picked mon.noname-a con 0x7feda41455c0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:11.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda41d7af0 con 0x7feda41455c0 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda4145a10 con 0x7feda41d8690 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40b5a20 con 0x7feda409ac40 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.727+0000 7fedaac93640 10 monclient(hunting): authenticate will time out at 769.039898s 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda3fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7feda41d8690 0x7feda4144eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60576/0 (socket says 172.21.15.129:60576) 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda3fff640 1 -- 172.21.15.129:0/1637942705 learned_addr learned my addr 172.21.15.129:0/1637942705 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 client.?.objecter ms_handle_connect 0x7feda41d8690 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 client.?.objecter ms_handle_connect 0x7feda41455c0 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 client.?.objecter ms_handle_connect 0x7feda409ac40 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772898003 0 0) 0x7feda4145a10 con 0x7feda41d8690 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed88004eb0 con 0x7feda41d8690 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646158670 0 0) 0x7feda41d7af0 con 0x7feda41455c0 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda4145a10 con 0x7feda41455c0 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927748489 0 0) 0x7feda40b5a20 con 0x7feda409ac40 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:57:11.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda41d7af0 con 0x7feda409ac40 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073115554 0 0) 0x7fed88004eb0 con 0x7feda41d8690 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda40b5a20 con 0x7feda41d8690 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fed90001860 con 0x7feda41d8690 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): dump: 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.723 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1714020580 0 0) 0x7feda4145a10 con 0x7feda41455c0 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed88004eb0 con 0x7feda41455c0 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fed9c001f00 con 0x7feda41455c0 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): dump: 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053366242 0 0) 0x7feda41d7af0 con 0x7feda409ac40 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda4145a10 con 0x7feda409ac40 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fed98001870 con 0x7feda409ac40 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient(hunting): dump: 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.725 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3855536583 0 0) 0x7feda40b5a20 con 0x7feda41d8690 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 >> v1:172.21.15.107:6790/0 conn(0x7feda409ac40 legacy=0x7feda41d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 >> v1:172.21.15.107:6789/0 conn(0x7feda41455c0 legacy=0x7feda41e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 monclient: found mon.b 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda4141960 con 0x7feda41d8690 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7fedaac93640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7fedaac93640 10 monclient: _renew_subs 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7fedaac93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7fedaac93640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda41e3b20 con 0x7feda41d8690 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7fedaac93640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7fedaac93640 10 monclient: _renew_subs 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7fedaac93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7fedaac93640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feda41e3e90 con 0x7feda41d8690 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7fedaac93640 1 librados: init done 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed90003610 con 0x7feda41d8690 2024-03-29T10:57:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fed90003a60 con 0x7feda41d8690 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda2ffd640 4 set_mon_vals no callback set 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 monclient: dump: 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fed90004010 con 0x7feda41d8690 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 804473273 0 0) 0x7fed90006630 con 0x7feda41d8690 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 10 client.4123.objecter ms_dispatch 0x7feda4000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.728+0000 7feda17fa640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.729+0000 7feda17fa640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.729+0000 7fedaac93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.729+0000 7fedaac93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.729+0000 7fedaac93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.729+0000 7fedaac93640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed84005080 con 0x7feda41d8690 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.731+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fed90006210 con 0x7feda41d8690 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.731+0000 7feda17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.731+0000 7feda17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:11.730 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:11.737+0000 7fc49e442040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:57:11.730 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:11.741+0000 7f1b61413040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:57:11.766 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:11.785+0000 7f1b61413040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:57:11.772 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:11.792+0000 7fc49e442040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:57:11.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.802+0000 7fedaac93640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e75d2a72-7727-4449-bcc3-d14b92fa0ce8", "id": 2}] 2024-03-29T10:57:11.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.802+0000 7fedaac93640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e75d2a72-7727-4449-bcc3-d14b92fa0ce8", "id": 2}] 2024-03-29T10:57:11.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.802+0000 7fedaac93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.802+0000 7fedaac93640 1 -- 172.21.15.129:0/1637942705 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e75d2a72-7727-4449-bcc3-d14b92fa0ce8", "id": 2} v 0) v1 -- 0x7fed840058b0 con 0x7feda41d8690 2024-03-29T10:57:11.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.806+0000 7feda17fa640 1 -- 172.21.15.129:0/1637942705 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e75d2a72-7727-4449-bcc3-d14b92fa0ce8", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2382180347 0 216599125) 0x7fed90006430 con 0x7feda41d8690 2024-03-29T10:57:11.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.806+0000 7feda17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e75d2a72-7727-4449-bcc3-d14b92fa0ce8", "id": 2}] 2024-03-29T10:57:11.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.806+0000 7feda17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:11.786 INFO:teuthology.orchestra.run.smithi129.stdout:2 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.806+0000 7fedaac93640 10 librados: watch_flush enter 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.806+0000 7fedaac93640 10 librados: watch_flush exit 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.806+0000 7fedaac93640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.806+0000 7fedaac93640 10 client.4123.objecter successfully canceled tick 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.807+0000 7fedaac93640 10 monclient: shutdown 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.807+0000 7fedaac93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.807+0000 7fedaac93640 1 -- 172.21.15.129:0/1637942705 >> v1:172.21.15.129:6789/0 conn(0x7feda41d8690 legacy=0x7feda4144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.807+0000 7fedaac93640 1 -- 172.21.15.129:0/1637942705 shutdown_connections 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.807+0000 7fedaac93640 1 -- 172.21.15.129:0/1637942705 >> 172.21.15.129:0/1637942705 conn(0x7feda4066b00 msgr2=0x7feda413a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.807+0000 7fedaac93640 1 -- 172.21.15.129:0/1637942705 shutdown_connections 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.807+0000 7fedaac93640 1 -- 172.21.15.129:0/1637942705 wait complete. 2024-03-29T10:57:11.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.807+0000 7fedaac93640 1 librados: shutdown 2024-03-29T10:57:11.797 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new a40725b6-723a-44b6-adde-604d836b6805 3 2024-03-29T10:57:11.820 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:11.840+0000 7f1b61413040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:57:11.820 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:11.840+0000 7fc49e442040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:57:11.862 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:11.882+0000 7f1b61413040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:57:11.877 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:11.897+0000 7fc49e442040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:57:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.917+0000 7f921435d640 10 monclient: get_monmap_and_config 2024-03-29T10:57:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.917+0000 7f921435d640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.917+0000 7f921435d640 10 monclient: monmap: 2024-03-29T10:57:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:57:11.918203+0000 2024-03-29T10:57:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:57:11.918203+0000 2024-03-29T10:57:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.898 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.899 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:11.899 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:11.899 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:11.899 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.917+0000 7f921435d640 1 Processor -- start 2024-03-29T10:57:11.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.917+0000 7f921435d640 1 -- start start 2024-03-29T10:57:11.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.917+0000 7f921435d640 10 monclient: init 2024-03-29T10:57:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 10 monclient(hunting): picked mon.noname-c con 0x7f920c09ac40 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 10 monclient(hunting): picked mon.noname-a con 0x7f920c09c870 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 10 monclient(hunting): picked mon.noname-b con 0x7f920c0b4360 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920c09dec0 con 0x7f920c09c870 2024-03-29T10:57:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920c09da30 con 0x7f920c09ac40 2024-03-29T10:57:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920c0b39c0 con 0x7f920c0b4360 2024-03-29T10:57:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f921435d640 10 monclient(hunting): authenticate will time out at 769.230682s 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f92120f9640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f920c09ac40 0x7f920c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60582/0 (socket says 172.21.15.129:60582) 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.918+0000 7f92120f9640 1 -- 172.21.15.129:0/3127838327 learned_addr learned my addr 172.21.15.129:0/3127838327 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307409574 0 0) 0x7f920c09da30 con 0x7f920c09ac40 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f8003520 con 0x7f920c09ac40 2024-03-29T10:57:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871161212 0 0) 0x7f920c0b39c0 con 0x7f920c0b4360 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c09da30 con 0x7f920c0b4360 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157720720 0 0) 0x7f91fc001110 con 0x7f920c09c870 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c0b39c0 con 0x7f920c09c870 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1367661232 0 0) 0x7f91f8003520 con 0x7f920c09ac40 2024-03-29T10:57:11.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 >> v1:172.21.15.107:6790/0 conn(0x7f920c0b4360 legacy=0x7f920c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 >> v1:172.21.15.107:6789/0 conn(0x7f920c09c870 legacy=0x7f920c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 monclient: found mon.noname-c 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c0b5160 con 0x7f920c09ac40 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9208001690 con 0x7f920c09ac40 2024-03-29T10:57:11.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f921435d640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f921435d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: dump: 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.912 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f921435d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9208001870 con 0x7f920c09ac40 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 1 -- 172.21.15.129:0/3127838327 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9208001c80 con 0x7f920c09ac40 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f921435d640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f921435d640 4 set_mon_vals no callback set 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f92110f7640 10 monclient: dump: 2024-03-29T10:57:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f921435d640 10 monclient: shutdown 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f921435d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.919+0000 7f921435d640 1 -- 172.21.15.129:0/3127838327 >> v1:172.21.15.129:6789/0 conn(0x7f920c09ac40 legacy=0x7f920c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.920+0000 7f921435d640 1 -- 172.21.15.129:0/3127838327 shutdown_connections 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.920+0000 7f921435d640 1 -- 172.21.15.129:0/3127838327 >> 172.21.15.129:0/3127838327 conn(0x7f920c066b00 msgr2=0x7f920c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.920+0000 7f921435d640 1 -- 172.21.15.129:0/3127838327 shutdown_connections 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.920+0000 7f921435d640 1 -- 172.21.15.129:0/3127838327 wait complete. 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.920+0000 7f921435d640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.920+0000 7f921435d640 10 monclient: monmap: 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.920+0000 7f921435d640 1 librados: starting msgr at 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.920+0000 7f921435d640 1 librados: starting objecter 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 1 Processor -- start 2024-03-29T10:57:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 1 -- start start 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 1 librados: setting wanted keys 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 1 librados: calling monclient init 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 10 monclient: init 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 10 monclient(hunting): picked mon.noname-a con 0x7f920c09ac40 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 10 monclient(hunting): picked mon.noname-c con 0x7f920c138f60 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 10 monclient(hunting): picked mon.noname-b con 0x7f920c0afdc0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920c1db610 con 0x7f920c09ac40 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920c139a00 con 0x7f920c0afdc0 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920c0b0230 con 0x7f920c138f60 2024-03-29T10:57:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f921435d640 10 monclient(hunting): authenticate will time out at 769.233552s 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f92128fa640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f920c0afdc0 0x7f920c1e1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60596/0 (socket says 172.21.15.129:60596) 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f92128fa640 1 -- 172.21.15.129:0/672541986 learned_addr learned my addr 172.21.15.129:0/672541986 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 client.?.objecter ms_handle_connect 0x7f920c0afdc0 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 client.?.objecter ms_handle_connect 0x7f920c09ac40 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 client.?.objecter ms_handle_connect 0x7f920c138f60 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327017602 0 0) 0x7f920c139a00 con 0x7f920c0afdc0 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f0004eb0 con 0x7f920c0afdc0 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087259518 0 0) 0x7f920c1db610 con 0x7f920c09ac40 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.921+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c139a00 con 0x7f920c09ac40 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563900259 0 0) 0x7f920c0b0230 con 0x7f920c138f60 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c1db610 con 0x7f920c138f60 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 316188183 0 0) 0x7f91f0004eb0 con 0x7f920c0afdc0 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f920c0b0230 con 0x7f920c0afdc0 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9204001c00 con 0x7f920c0afdc0 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3840753572 0 0) 0x7f920c139a00 con 0x7f920c09ac40 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91f0004eb0 con 0x7f920c09ac40 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9208001b80 con 0x7f920c09ac40 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1948261968 0 0) 0x7f920c1db610 con 0x7f920c138f60 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f920c139a00 con 0x7f920c138f60 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f91fc001860 con 0x7f920c138f60 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:11.922 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2205194222 0 0) 0x7f920c0b0230 con 0x7f920c0afdc0 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 >> v1:172.21.15.107:6790/0 conn(0x7f920c138f60 legacy=0x7f920c0af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 >> v1:172.21.15.107:6789/0 conn(0x7f920c09ac40 legacy=0x7f920c1dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 monclient: found mon.b 2024-03-29T10:57:11.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920c0b03c0 con 0x7f920c0afdc0 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f921435d640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f921435d640 10 monclient: _renew_subs 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f921435d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f921435d640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920c1e36e0 con 0x7f920c0afdc0 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f921435d640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f921435d640 10 monclient: _renew_subs 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9204003990 con 0x7f920c0afdc0 2024-03-29T10:57:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f921435d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f921435d640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f920c1e3a50 con 0x7f920c0afdc0 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f921435d640 1 librados: init done 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9204003b80 con 0x7f920c0afdc0 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 monclient: dump: 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:11.926 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f92108f6640 4 set_mon_vals no callback set 2024-03-29T10:57:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9204004070 con 0x7f920c0afdc0 2024-03-29T10:57:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3441819584 0 0) 0x7f92040047b0 con 0x7f920c0afdc0 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 10 client.4129.objecter ms_dispatch 0x7f920c000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.922+0000 7f9202ffd640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.923+0000 7f921435d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.923+0000 7f921435d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.923+0000 7f921435d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.923+0000 7f921435d640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f4005080 con 0x7f920c0afdc0 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.925+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9204005b40 con 0x7f920c0afdc0 2024-03-29T10:57:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.925+0000 7f9202ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.925+0000 7f9202ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:11.928 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:11.942+0000 7fc49e442040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:57:11.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.997+0000 7f921435d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a40725b6-723a-44b6-adde-604d836b6805", "id": 3}] 2024-03-29T10:57:11.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.997+0000 7f921435d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a40725b6-723a-44b6-adde-604d836b6805", "id": 3}] 2024-03-29T10:57:11.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.997+0000 7f921435d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:11.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:11.997+0000 7f921435d640 1 -- 172.21.15.129:0/672541986 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a40725b6-723a-44b6-adde-604d836b6805", "id": 3} v 0) v1 -- 0x7f91f40058b0 con 0x7f920c0afdc0 2024-03-29T10:57:11.978 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:11.997+0000 7f1b61413040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:57:11.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.000+0000 7f9202ffd640 1 -- 172.21.15.129:0/672541986 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a40725b6-723a-44b6-adde-604d836b6805", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 114226547 0 525046306) 0x7f9204004320 con 0x7f920c0afdc0 2024-03-29T10:57:11.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f9202ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a40725b6-723a-44b6-adde-604d836b6805", "id": 3}] 2024-03-29T10:57:11.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f9202ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:11.981 INFO:teuthology.orchestra.run.smithi129.stdout:3 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 10 librados: watch_flush enter 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 10 librados: watch_flush exit 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 10 client.4129.objecter successfully canceled tick 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 10 monclient: shutdown 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 1 -- 172.21.15.129:0/672541986 >> v1:172.21.15.129:6789/0 conn(0x7f920c0afdc0 legacy=0x7f920c1e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 1 -- 172.21.15.129:0/672541986 shutdown_connections 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 1 -- 172.21.15.129:0/672541986 >> 172.21.15.129:0/672541986 conn(0x7f920c066b00 msgr2=0x7f920c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 1 -- 172.21.15.129:0/672541986 shutdown_connections 2024-03-29T10:57:11.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.001+0000 7f921435d640 1 -- 172.21.15.129:0/672541986 wait complete. 2024-03-29T10:57:11.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.002+0000 7f921435d640 1 librados: shutdown 2024-03-29T10:57:11.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e 4 2024-03-29T10:57:12.042 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:12.062+0000 7fc49e442040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:57:12.048 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:12.067+0000 7f1b61413040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:57:12.093 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:12.113+0000 7f1b61413040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:57:12.116 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:12.135+0000 7fc49e442040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:57:12.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.151+0000 7f3c46b3d640 10 monclient: get_monmap_and_config 2024-03-29T10:57:12.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.151+0000 7f3c46b3d640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.151+0000 7f3c46b3d640 10 monclient: monmap: 2024-03-29T10:57:12.132 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:12.132 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.132 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:57:12.152506+0000 2024-03-29T10:57:12.132 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:57:12.152506+0000 2024-03-29T10:57:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:12.133 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:12.133 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 1 Processor -- start 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 1 -- start start 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 10 monclient: init 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f3c4009ac40 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f3c4009c870 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f3c400b4360 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4009dec0 con 0x7f3c4009c870 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4009da30 con 0x7f3c4009ac40 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c400b39c0 con 0x7f3c400b4360 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.152+0000 7f3c46b3d640 10 monclient(hunting): authenticate will time out at 769.464888s 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c448d9640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3c4009ac40 0x7f3c400b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60604/0 (socket says 172.21.15.129:60604) 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c448d9640 1 -- 172.21.15.129:0/2968410017 learned_addr learned my addr 172.21.15.129:0/2968410017 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437556383 0 0) 0x7f3c4009da30 con 0x7f3c4009ac40 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c30003520 con 0x7f3c4009ac40 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704169262 0 0) 0x7f3c400b39c0 con 0x7f3c400b4360 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:57:12.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c4009da30 con 0x7f3c400b4360 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63881464 0 0) 0x7f3c2c001110 con 0x7f3c4009c870 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c400b39c0 con 0x7f3c4009c870 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2607729273 0 0) 0x7f3c30003520 con 0x7f3c4009ac40 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 >> v1:172.21.15.107:6790/0 conn(0x7f3c400b4360 legacy=0x7f3c4014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 >> v1:172.21.15.107:6789/0 conn(0x7f3c4009c870 legacy=0x7f3c4013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 monclient: found mon.noname-c 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c400b5160 con 0x7f3c4009ac40 2024-03-29T10:57:12.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3c34001690 con 0x7f3c4009ac40 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c46b3d640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c46b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: dump: 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c46b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c34001870 con 0x7f3c4009ac40 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c46b3d640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 1 -- 172.21.15.129:0/2968410017 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3c34001c80 con 0x7f3c4009ac40 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c46b3d640 4 set_mon_vals no callback set 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.153+0000 7f3c3f7fe640 10 monclient: dump: 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.146 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.154+0000 7f3c46b3d640 10 monclient: shutdown 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.154+0000 7f3c46b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.154+0000 7f3c46b3d640 1 -- 172.21.15.129:0/2968410017 >> v1:172.21.15.129:6789/0 conn(0x7f3c4009ac40 legacy=0x7f3c400b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.154+0000 7f3c46b3d640 1 -- 172.21.15.129:0/2968410017 shutdown_connections 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.154+0000 7f3c46b3d640 1 -- 172.21.15.129:0/2968410017 >> 172.21.15.129:0/2968410017 conn(0x7f3c40066b00 msgr2=0x7f3c40066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.154+0000 7f3c46b3d640 1 -- 172.21.15.129:0/2968410017 shutdown_connections 2024-03-29T10:57:12.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.154+0000 7f3c46b3d640 1 -- 172.21.15.129:0/2968410017 wait complete. 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.154+0000 7f3c46b3d640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.154+0000 7f3c46b3d640 10 monclient: monmap: 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 1 librados: starting msgr at 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 1 librados: starting objecter 2024-03-29T10:57:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 1 Processor -- start 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 1 -- start start 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 1 librados: setting wanted keys 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 1 librados: calling monclient init 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 10 monclient: init 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f3c4009ac40 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f3c40138f60 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f3c400afdc0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c401db610 con 0x7f3c40138f60 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.155+0000 7f3c46b3d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c40139a00 con 0x7f3c4009ac40 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c400b0230 con 0x7f3c400afdc0 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 10 monclient(hunting): authenticate will time out at 769.467936s 2024-03-29T10:57:12.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c448d9640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f3c4009ac40 0x7f3c401dac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60614/0 (socket says 172.21.15.129:60614) 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c448d9640 1 -- 172.21.15.129:0/3294947761 learned_addr learned my addr 172.21.15.129:0/3294947761 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 client.?.objecter ms_handle_connect 0x7f3c4009ac40 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 client.?.objecter ms_handle_connect 0x7f3c400afdc0 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425654512 0 0) 0x7f3c40139a00 con 0x7f3c4009ac40 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c24004eb0 con 0x7f3c4009ac40 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 client.?.objecter ms_handle_connect 0x7f3c40138f60 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890024469 0 0) 0x7f3c400b0230 con 0x7f3c400afdc0 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c40139a00 con 0x7f3c400afdc0 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250448156 0 0) 0x7f3c401db610 con 0x7f3c40138f60 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c400b0230 con 0x7f3c40138f60 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1116429012 0 0) 0x7f3c24004eb0 con 0x7f3c4009ac40 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c401db610 con 0x7f3c4009ac40 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3c34001bc0 con 0x7f3c4009ac40 2024-03-29T10:57:12.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): dump: 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1003987789 0 0) 0x7f3c400b0230 con 0x7f3c40138f60 2024-03-29T10:57:12.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c24004eb0 con 0x7f3c40138f60 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534790224 0 0) 0x7f3c40139a00 con 0x7f3c400afdc0 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c400b0230 con 0x7f3c400afdc0 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3711151221 0 0) 0x7f3c401db610 con 0x7f3c4009ac40 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 >> v1:172.21.15.107:6790/0 conn(0x7f3c400afdc0 legacy=0x7f3c401e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 >> v1:172.21.15.107:6789/0 conn(0x7f3c40138f60 legacy=0x7f3c400af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 monclient: found mon.b 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c400b03c0 con 0x7f3c4009ac40 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:57:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 10 monclient: _renew_subs 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c401e36e0 con 0x7f3c4009ac40 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 10 monclient: _renew_subs 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c401e3a50 con 0x7f3c4009ac40 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c46b3d640 1 librados: init done 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c34004bf0 con 0x7f3c4009ac40 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3c34004d60 con 0x7f3c4009ac40 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 10 monclient: dump: 2024-03-29T10:57:12.154 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3c34005250 con 0x7f3c4009ac40 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.156+0000 7f3c3effd640 4 set_mon_vals no callback set 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.157+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3054582552 0 0) 0x7f3c340067b0 con 0x7f3c4009ac40 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.157+0000 7f3c3d7fa640 10 client.4135.objecter ms_dispatch 0x7f3c40000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.157+0000 7f3c3d7fa640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:57:12.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.157+0000 7f3c3d7fa640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:57:12.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.157+0000 7f3c3d7fa640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T10:57:12.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.157+0000 7f3c46b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.157+0000 7f3c46b3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.158+0000 7f3c46b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.158+0000 7f3c46b3d640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c20005080 con 0x7f3c4009ac40 2024-03-29T10:57:12.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.159+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3c340062a0 con 0x7f3c4009ac40 2024-03-29T10:57:12.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.159+0000 7f3c3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.159+0000 7f3c3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:12.156 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:12.165+0000 7f1b61413040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:57:12.164 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:12.184+0000 7fc49e442040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:57:12.192 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:12.212+0000 7f1b61413040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:57:12.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.230+0000 7f3c46b3d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e", "id": 4}] 2024-03-29T10:57:12.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.230+0000 7f3c46b3d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e", "id": 4}] 2024-03-29T10:57:12.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.230+0000 7f3c46b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.230+0000 7f3c46b3d640 1 -- 172.21.15.129:0/3294947761 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e", "id": 4} v 0) v1 -- 0x7f3c200058b0 con 0x7f3c4009ac40 2024-03-29T10:57:12.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.234+0000 7f3c3d7fa640 1 -- 172.21.15.129:0/3294947761 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1939533375 0 1697012583) 0x7f3c34006510 con 0x7f3c4009ac40 2024-03-29T10:57:12.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.234+0000 7f3c3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e", "id": 4}] 2024-03-29T10:57:12.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.234+0000 7f3c3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:12.215 INFO:teuthology.orchestra.run.smithi129.stdout:4 2024-03-29T10:57:12.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 10 librados: watch_flush enter 2024-03-29T10:57:12.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 10 librados: watch_flush exit 2024-03-29T10:57:12.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T10:57:12.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 10 client.4135.objecter successfully canceled tick 2024-03-29T10:57:12.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 10 monclient: shutdown 2024-03-29T10:57:12.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:12.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 1 -- 172.21.15.129:0/3294947761 >> v1:172.21.15.129:6789/0 conn(0x7f3c4009ac40 legacy=0x7f3c401dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 1 -- 172.21.15.129:0/3294947761 shutdown_connections 2024-03-29T10:57:12.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 1 -- 172.21.15.129:0/3294947761 >> 172.21.15.129:0/3294947761 conn(0x7f3c40066b00 msgr2=0x7f3c400b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:12.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 1 -- 172.21.15.129:0/3294947761 shutdown_connections 2024-03-29T10:57:12.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 1 -- 172.21.15.129:0/3294947761 wait complete. 2024-03-29T10:57:12.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.235+0000 7f3c46b3d640 1 librados: shutdown 2024-03-29T10:57:12.217 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:12.237+0000 7fc49e442040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:57:12.226 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 641286c5-474a-40d0-9b3f-a8fec463271f 5 2024-03-29T10:57:12.266 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:12.286+0000 7fc49e442040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:57:12.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.384+0000 7f6cd034d640 10 monclient: get_monmap_and_config 2024-03-29T10:57:12.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.384+0000 7f6cd034d640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.384+0000 7f6cd034d640 10 monclient: monmap: 2024-03-29T10:57:12.414 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:12.414 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:57:12.385717+0000 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:57:12.385717+0000 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.385+0000 7f6cd034d640 1 Processor -- start 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.385+0000 7f6cd034d640 1 -- start start 2024-03-29T10:57:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.385+0000 7f6cd034d640 10 monclient: init 2024-03-29T10:57:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 10 monclient(hunting): picked mon.noname-b con 0x7f6cc809ac40 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 10 monclient(hunting): picked mon.noname-c con 0x7f6cc809c870 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 10 monclient(hunting): picked mon.noname-a con 0x7f6cc80b4360 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc809dec0 con 0x7f6cc80b4360 2024-03-29T10:57:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc809da30 con 0x7f6cc809c870 2024-03-29T10:57:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc80b39c0 con 0x7f6cc809ac40 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 10 monclient(hunting): authenticate will time out at 769.698150s 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd8e8640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f6cc809c870 0x7f6cc813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60630/0 (socket says 172.21.15.129:60630) 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd8e8640 1 -- 172.21.15.129:0/2824958594 learned_addr learned my addr 172.21.15.129:0/2824958594 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72225826 0 0) 0x7f6cc809da30 con 0x7f6cc809c870 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb4003520 con 0x7f6cc809c870 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752048031 0 0) 0x7f6cc0001490 con 0x7f6cc80b4360 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc809da30 con 0x7f6cc80b4360 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040319435 0 0) 0x7f6cc80b39c0 con 0x7f6cc809ac40 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc0001490 con 0x7f6cc809ac40 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187414821 0 0) 0x7f6cb4003520 con 0x7f6cc809c870 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 >> v1:172.21.15.107:6790/0 conn(0x7f6cc809ac40 legacy=0x7f6cc80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 >> v1:172.21.15.107:6789/0 conn(0x7f6cc80b4360 legacy=0x7f6cc814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 monclient: found mon.noname-c 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc80b5160 con 0x7f6cc809c870 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6cc4001690 con 0x7f6cc809c870 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: dump: 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.428 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cc4001870 con 0x7f6cc809c870 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 1 -- 172.21.15.129:0/2824958594 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6cc4001c80 con 0x7f6cc809c870 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6cd034d640 4 set_mon_vals no callback set 2024-03-29T10:57:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.386+0000 7f6ccd0e7640 10 monclient: dump: 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.387+0000 7f6cd034d640 10 monclient: shutdown 2024-03-29T10:57:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.387+0000 7f6cd034d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.387+0000 7f6cd034d640 1 -- 172.21.15.129:0/2824958594 >> v1:172.21.15.129:6789/0 conn(0x7f6cc809c870 legacy=0x7f6cc813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.387+0000 7f6cd034d640 1 -- 172.21.15.129:0/2824958594 shutdown_connections 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.387+0000 7f6cd034d640 1 -- 172.21.15.129:0/2824958594 >> 172.21.15.129:0/2824958594 conn(0x7f6cc8066b00 msgr2=0x7f6cc8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.387+0000 7f6cd034d640 1 -- 172.21.15.129:0/2824958594 shutdown_connections 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.387+0000 7f6cd034d640 1 -- 172.21.15.129:0/2824958594 wait complete. 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient: monmap: 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 1 librados: starting msgr at 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 1 librados: starting objecter 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 1 Processor -- start 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 1 -- start start 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 1 librados: setting wanted keys 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 1 librados: calling monclient init 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient: init 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient(hunting): picked mon.noname-c con 0x7f6cc809ac40 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient(hunting): picked mon.noname-b con 0x7f6cc80af4e0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient(hunting): picked mon.noname-a con 0x7f6cc81d7cf0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc80b1940 con 0x7f6cc81d7cf0 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc81d70c0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc80acd00 con 0x7f6cc809ac40 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.388+0000 7f6cd034d640 10 monclient(hunting): authenticate will time out at 769.700908s 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6ccd8e8640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f6cc80af4e0 0x7f6cc81d75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60636/0 (socket says 172.21.15.129:60636) 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6ccd8e8640 1 -- 172.21.15.129:0/730535491 learned_addr learned my addr 172.21.15.129:0/730535491 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 client.?.objecter ms_handle_connect 0x7f6cc80af4e0 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 client.?.objecter ms_handle_connect 0x7f6cc81d7cf0 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 client.?.objecter ms_handle_connect 0x7f6cc809ac40 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211637291 0 0) 0x7f6cc81d70c0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cac004eb0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542262518 0 0) 0x7f6cc80b1940 con 0x7f6cc81d7cf0 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc81d70c0 con 0x7f6cc81d7cf0 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228101491 0 0) 0x7f6cc80acd00 con 0x7f6cc809ac40 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc80b1940 con 0x7f6cc809ac40 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3738768945 0 0) 0x7f6cac004eb0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cc80acd00 con 0x7f6cc80af4e0 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6cc4001bc0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): dump: 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4202051515 0 0) 0x7f6cc81d70c0 con 0x7f6cc81d7cf0 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cac004eb0 con 0x7f6cc81d7cf0 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6cc0001a00 con 0x7f6cc81d7cf0 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): dump: 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2998711930 0 0) 0x7f6cc80b1940 con 0x7f6cc809ac40 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cc81d70c0 con 0x7f6cc809ac40 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6cb8001830 con 0x7f6cc809ac40 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient(hunting): dump: 2024-03-29T10:57:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 457239679 0 0) 0x7f6cc80acd00 con 0x7f6cc80af4e0 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 >> v1:172.21.15.107:6790/0 conn(0x7f6cc809ac40 legacy=0x7f6cc80b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 >> v1:172.21.15.107:6789/0 conn(0x7f6cc81d7cf0 legacy=0x7f6cc81e1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 monclient: found mon.b 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc80af970 con 0x7f6cc80af4e0 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cd034d640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cd034d640 10 monclient: _renew_subs 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cd034d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cd034d640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc81e3260 con 0x7f6cc80af4e0 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cd034d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cd034d640 10 monclient: _renew_subs 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cd034d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cd034d640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc81e35d0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cd034d640 1 librados: init done 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cc4004bf0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6cbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.389+0000 7f6ccc8e6640 4 set_mon_vals no callback set 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6cc4004f00 con 0x7f6cc80af4e0 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 10 monclient: dump: 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6cc40053f0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2862730509 0 0) 0x7f6cc4006990 con 0x7f6cc80af4e0 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 10 client.4141.objecter ms_dispatch 0x7f6cc8000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cd034d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cbeffd640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cd034d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cd034d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.390+0000 7f6cd034d640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb0005080 con 0x7f6cc80af4e0 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.392+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6cc4006ba0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.392+0000 7f6cbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.392+0000 7f6cbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:12.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.463+0000 7f6cd034d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "641286c5-474a-40d0-9b3f-a8fec463271f", "id": 5}] 2024-03-29T10:57:12.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.463+0000 7f6cd034d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "641286c5-474a-40d0-9b3f-a8fec463271f", "id": 5}] 2024-03-29T10:57:12.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.463+0000 7f6cd034d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.463+0000 7f6cd034d640 1 -- 172.21.15.129:0/730535491 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "641286c5-474a-40d0-9b3f-a8fec463271f", "id": 5} v 0) v1 -- 0x7f6cb00058b0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.467+0000 7f6cbeffd640 1 -- 172.21.15.129:0/730535491 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "641286c5-474a-40d0-9b3f-a8fec463271f", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3107900649 0 1988412176) 0x7f6cc40064e0 con 0x7f6cc80af4e0 2024-03-29T10:57:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.467+0000 7f6cbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "641286c5-474a-40d0-9b3f-a8fec463271f", "id": 5}] 2024-03-29T10:57:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.467+0000 7f6cbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:12.448 INFO:teuthology.orchestra.run.smithi129.stdout:5 2024-03-29T10:57:12.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 10 librados: watch_flush enter 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 10 librados: watch_flush exit 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 10 client.4141.objecter successfully canceled tick 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 10 monclient: shutdown 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 1 -- 172.21.15.129:0/730535491 >> v1:172.21.15.129:6789/0 conn(0x7f6cc80af4e0 legacy=0x7f6cc81d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 1 -- 172.21.15.129:0/730535491 shutdown_connections 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 1 -- 172.21.15.129:0/730535491 >> 172.21.15.129:0/730535491 conn(0x7f6cc8066b00 msgr2=0x7f6cc813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 1 -- 172.21.15.129:0/730535491 shutdown_connections 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 1 -- 172.21.15.129:0/730535491 wait complete. 2024-03-29T10:57:12.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.468+0000 7f6cd034d640 1 librados: shutdown 2024-03-29T10:57:12.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 22d61d7c-904b-4a1e-9555-7929b313179e 6 2024-03-29T10:57:12.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.580+0000 7fc51d6b3640 10 monclient: get_monmap_and_config 2024-03-29T10:57:12.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.580+0000 7fc51d6b3640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.580+0000 7fc51d6b3640 10 monclient: monmap: 2024-03-29T10:57:12.561 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:12.561 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.561 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:57:12.581715+0000 2024-03-29T10:57:12.561 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:57:12.581715+0000 2024-03-29T10:57:12.561 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.562 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.562 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:12.562 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:12.562 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:12.562 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.581+0000 7fc51d6b3640 1 Processor -- start 2024-03-29T10:57:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.581+0000 7fc51d6b3640 1 -- start start 2024-03-29T10:57:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.581+0000 7fc51d6b3640 10 monclient: init 2024-03-29T10:57:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.581+0000 7fc51d6b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:12.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 10 monclient(hunting): picked mon.noname-a con 0x7fc5180b8240 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 10 monclient(hunting): picked mon.noname-c con 0x7fc5180b1f60 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 10 monclient(hunting): picked mon.noname-b con 0x7fc51809c8c0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51809bd80 con 0x7fc5180b8240 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51809b960 con 0x7fc5180b1f60 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5180b2ec0 con 0x7fc51809c8c0 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 10 monclient(hunting): authenticate will time out at 769.894025s 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc5167fc640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc5180b1f60 0x7fc51809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60642/0 (socket says 172.21.15.129:60642) 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc5167fc640 1 -- 172.21.15.129:0/3526446951 learned_addr learned my addr 172.21.15.129:0/3526446951 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948155071 0 0) 0x7fc51809b960 con 0x7fc5180b1f60 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc500003520 con 0x7fc5180b1f60 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332313782 0 0) 0x7fc51809bd80 con 0x7fc5180b8240 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc51809b960 con 0x7fc5180b8240 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851222372 0 0) 0x7fc5180b2ec0 con 0x7fc51809c8c0 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc51809bd80 con 0x7fc51809c8c0 2024-03-29T10:57:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3039909324 0 0) 0x7fc500003520 con 0x7fc5180b1f60 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 >> v1:172.21.15.107:6790/0 conn(0x7fc51809c8c0 legacy=0x7fc51814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 >> v1:172.21.15.107:6789/0 conn(0x7fc5180b8240 legacy=0x7fc51813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 monclient: found mon.noname-c 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51813fb60 con 0x7fc5180b1f60 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc508001690 con 0x7fc5180b1f60 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: dump: 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc508001870 con 0x7fc5180b1f60 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 1 -- 172.21.15.129:0/3526446951 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc508001c80 con 0x7fc5180b1f60 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc515ffb640 10 monclient: dump: 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.582+0000 7fc51d6b3640 4 set_mon_vals no callback set 2024-03-29T10:57:12.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.583+0000 7fc51d6b3640 10 monclient: shutdown 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.583+0000 7fc51d6b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.583+0000 7fc51d6b3640 1 -- 172.21.15.129:0/3526446951 >> v1:172.21.15.129:6789/0 conn(0x7fc5180b1f60 legacy=0x7fc51809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.583+0000 7fc51d6b3640 1 -- 172.21.15.129:0/3526446951 shutdown_connections 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.583+0000 7fc51d6b3640 1 -- 172.21.15.129:0/3526446951 >> 172.21.15.129:0/3526446951 conn(0x7fc518066b00 msgr2=0x7fc518066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.583+0000 7fc51d6b3640 1 -- 172.21.15.129:0/3526446951 shutdown_connections 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.583+0000 7fc51d6b3640 1 -- 172.21.15.129:0/3526446951 wait complete. 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.584+0000 7fc51d6b3640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.584+0000 7fc51d6b3640 10 monclient: monmap: 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-03-29T10:57:12.577 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.584+0000 7fc51d6b3640 1 librados: starting msgr at 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.584+0000 7fc51d6b3640 1 librados: starting objecter 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.584+0000 7fc51d6b3640 1 Processor -- start 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.584+0000 7fc51d6b3640 1 -- start start 2024-03-29T10:57:12.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.584+0000 7fc51d6b3640 1 librados: setting wanted keys 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.584+0000 7fc51d6b3640 1 librados: calling monclient init 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.584+0000 7fc51d6b3640 10 monclient: init 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 10 monclient(hunting): picked mon.noname-c con 0x7fc51809c8c0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 10 monclient(hunting): picked mon.noname-a con 0x7fc5180b8240 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 10 monclient(hunting): picked mon.noname-b con 0x7fc518144bf0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc518131f80 con 0x7fc5180b8240 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc518133010 con 0x7fc518144bf0 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5181332f0 con 0x7fc51809c8c0 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 10 monclient(hunting): authenticate will time out at 769.897092s 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc516ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc51809c8c0 0x7fc518131720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.129:46956/0 (socket says 172.21.15.129:46956) 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc5177fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc518144bf0 0x7fc518145300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60644/0 (socket says 172.21.15.129:60644) 2024-03-29T10:57:12.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc516ffd640 1 -- 172.21.15.129:0/1931238069 learned_addr learned my addr 172.21.15.129:0/1931238069 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc5167fc640 1 --1- 172.21.15.129:0/1931238069 >> v1:172.21.15.107:6789/0 conn(0x7fc5180b8240 0x7fc5181444e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.129:44626/0 (socket says 172.21.15.129:44626) 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 client.?.objecter ms_handle_connect 0x7fc518144bf0 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 client.?.objecter ms_handle_connect 0x7fc5180b8240 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 client.?.objecter ms_handle_connect 0x7fc51809c8c0 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862538443 0 0) 0x7fc518133010 con 0x7fc518144bf0 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc504004eb0 con 0x7fc518144bf0 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316233041 0 0) 0x7fc5181332f0 con 0x7fc51809c8c0 2024-03-29T10:57:12.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc518133010 con 0x7fc51809c8c0 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015848412 0 0) 0x7fc518131f80 con 0x7fc5180b8240 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5181332f0 con 0x7fc5180b8240 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3879099639 0 0) 0x7fc504004eb0 con 0x7fc518144bf0 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc518131f80 con 0x7fc518144bf0 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc510001bd0 con 0x7fc518144bf0 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1094726219 0 0) 0x7fc518133010 con 0x7fc51809c8c0 2024-03-29T10:57:12.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc504004eb0 con 0x7fc51809c8c0 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 205404194 0 0) 0x7fc5181332f0 con 0x7fc5180b8240 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc518133010 con 0x7fc5180b8240 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc50c001870 con 0x7fc51809c8c0 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.583 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc5080019e0 con 0x7fc5180b8240 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1495705468 0 0) 0x7fc518131f80 con 0x7fc518144bf0 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 >> v1:172.21.15.107:6790/0 conn(0x7fc51809c8c0 legacy=0x7fc518131720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 >> v1:172.21.15.107:6789/0 conn(0x7fc5180b8240 legacy=0x7fc5181444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 monclient: found mon.b 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc518132410 con 0x7fc518144bf0 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc4ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 10 monclient: _renew_subs 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.585+0000 7fc51d6b3640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc518131190 con 0x7fc518144bf0 2024-03-29T10:57:12.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc51d6b3640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc51d6b3640 10 monclient: _renew_subs 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc51d6b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc51d6b3640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc518145a10 con 0x7fc518144bf0 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc51d6b3640 1 librados: init done 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5100020c0 con 0x7fc518144bf0 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc510003b30 con 0x7fc518144bf0 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 10 monclient: dump: 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.586 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc5157fa640 4 set_mon_vals no callback set 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc5100040e0 con 0x7fc518144bf0 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4001301855 0 0) 0x7fc510006ad0 con 0x7fc518144bf0 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 10 client.4147.objecter ms_dispatch 0x7fc518000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc4ff7fe640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc51d6b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc51d6b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc51d6b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.586+0000 7fc51d6b3640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f8005080 con 0x7fc518144bf0 2024-03-29T10:57:12.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.588+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc510004670 con 0x7fc518144bf0 2024-03-29T10:57:12.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.588+0000 7fc4ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.588+0000 7fc4ff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:12.589 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:12.594+0000 7f1b61413040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:57:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.660+0000 7fc51d6b3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "22d61d7c-904b-4a1e-9555-7929b313179e", "id": 6}] 2024-03-29T10:57:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.660+0000 7fc51d6b3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "22d61d7c-904b-4a1e-9555-7929b313179e", "id": 6}] 2024-03-29T10:57:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.660+0000 7fc51d6b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.660+0000 7fc51d6b3640 1 -- 172.21.15.129:0/1931238069 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "22d61d7c-904b-4a1e-9555-7929b313179e", "id": 6} v 0) v1 -- 0x7fc4f80058b0 con 0x7fc518144bf0 2024-03-29T10:57:12.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.664+0000 7fc4ff7fe640 1 -- 172.21.15.129:0/1931238069 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "22d61d7c-904b-4a1e-9555-7929b313179e", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2413854523 0 1113811849) 0x7fc5100064e0 con 0x7fc518144bf0 2024-03-29T10:57:12.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.664+0000 7fc4ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "22d61d7c-904b-4a1e-9555-7929b313179e", "id": 6}] 2024-03-29T10:57:12.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.664+0000 7fc4ff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:12.645 INFO:teuthology.orchestra.run.smithi129.stdout:6 2024-03-29T10:57:12.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.664+0000 7fc51d6b3640 10 librados: watch_flush enter 2024-03-29T10:57:12.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.664+0000 7fc51d6b3640 10 librados: watch_flush exit 2024-03-29T10:57:12.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T10:57:12.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 10 client.4147.objecter successfully canceled tick 2024-03-29T10:57:12.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 10 monclient: shutdown 2024-03-29T10:57:12.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:12.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 1 -- 172.21.15.129:0/1931238069 >> v1:172.21.15.129:6789/0 conn(0x7fc518144bf0 legacy=0x7fc518145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 1 -- 172.21.15.129:0/1931238069 shutdown_connections 2024-03-29T10:57:12.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 1 -- 172.21.15.129:0/1931238069 >> 172.21.15.129:0/1931238069 conn(0x7fc518066b00 msgr2=0x7fc51813dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:12.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 1 -- 172.21.15.129:0/1931238069 shutdown_connections 2024-03-29T10:57:12.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 1 -- 172.21.15.129:0/1931238069 wait complete. 2024-03-29T10:57:12.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.665+0000 7fc51d6b3640 1 librados: shutdown 2024-03-29T10:57:12.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new d8e2a8ab-8a6e-4781-b029-8c6277133458 7 2024-03-29T10:57:12.665 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:12.685+0000 7fc49e442040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:57:12.709 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:12.728+0000 7f1b61413040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:57:12.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.776+0000 7f4ee4fa3640 10 monclient: get_monmap_and_config 2024-03-29T10:57:12.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.776+0000 7f4ee4fa3640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.776+0000 7f4ee4fa3640 10 monclient: monmap: 2024-03-29T10:57:12.757 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:12.757 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.757 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:57:12.777574+0000 2024-03-29T10:57:12.757 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:57:12.777574+0000 2024-03-29T10:57:12.758 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.758 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.758 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:12.758 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:12.758 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:12.758 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.777+0000 7f4ee4fa3640 1 Processor -- start 2024-03-29T10:57:12.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.777+0000 7f4ee4fa3640 1 -- start start 2024-03-29T10:57:12.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.777+0000 7f4ee4fa3640 10 monclient: init 2024-03-29T10:57:12.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.777+0000 7f4ee4fa3640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.777+0000 7f4ee4fa3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:12.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f4ee009ac40 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:12.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f4ee009c870 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f4ee00b4360 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee009dec0 con 0x7f4ee00b4360 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee009da30 con 0x7f4ee009ac40 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee00b39c0 con 0x7f4ee009c870 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 10 monclient(hunting): authenticate will time out at 770.089984s 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ede59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4ee009ac40 0x7f4ee00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60660/0 (socket says 172.21.15.129:60660) 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ede59c640 1 -- 172.21.15.129:0/1558463142 learned_addr learned my addr 172.21.15.129:0/1558463142 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504473973 0 0) 0x7f4ee009da30 con 0x7f4ee009ac40 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:57:12.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec8003520 con 0x7f4ee009ac40 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168971919 0 0) 0x7f4ee00b39c0 con 0x7f4ee009c870 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee009da30 con 0x7f4ee009c870 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911868573 0 0) 0x7f4ed8001490 con 0x7f4ee00b4360 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee00b39c0 con 0x7f4ee00b4360 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387150600 0 0) 0x7f4ec8003520 con 0x7f4ee009ac40 2024-03-29T10:57:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 >> v1:172.21.15.107:6790/0 conn(0x7f4ee009c870 legacy=0x7f4ee013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 >> v1:172.21.15.107:6789/0 conn(0x7f4ee00b4360 legacy=0x7f4ee014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 monclient: found mon.noname-c 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee00b5160 con 0x7f4ee009ac40 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4ed4001690 con 0x7f4ee009ac40 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: dump: 2024-03-29T10:57:12.769 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ed40017f0 con 0x7f4ee009ac40 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 1 -- 172.21.15.129:0/1558463142 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4ed4001c00 con 0x7f4ee009ac40 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:12.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4ee4fa3640 4 set_mon_vals no callback set 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.778+0000 7f4edd59a640 10 monclient: dump: 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.779+0000 7f4ee4fa3640 10 monclient: shutdown 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.779+0000 7f4ee4fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.779+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/1558463142 >> v1:172.21.15.129:6789/0 conn(0x7f4ee009ac40 legacy=0x7f4ee00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.779+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/1558463142 shutdown_connections 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.779+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/1558463142 >> 172.21.15.129:0/1558463142 conn(0x7f4ee0066b00 msgr2=0x7f4ee0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.779+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/1558463142 shutdown_connections 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.779+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/1558463142 wait complete. 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.780+0000 7f4ee4fa3640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.780+0000 7f4ee4fa3640 10 monclient: monmap: 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 0 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 0.000000 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:created 0.000000 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.772 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.780+0000 7f4ee4fa3640 1 librados: starting msgr at 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.780+0000 7f4ee4fa3640 1 librados: starting objecter 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.780+0000 7f4ee4fa3640 1 Processor -- start 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.780+0000 7f4ee4fa3640 1 -- start start 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.780+0000 7f4ee4fa3640 1 librados: setting wanted keys 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.780+0000 7f4ee4fa3640 1 librados: calling monclient init 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.780+0000 7f4ee4fa3640 10 monclient: init 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f4ee009ac40 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f4ee0138f70 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:12.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f4ee00afe80 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee01d8080 con 0x7f4ee00afe80 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee00b1af0 con 0x7f4ee009ac40 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee00b02f0 con 0x7f4ee0138f70 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ee4fa3640 10 monclient(hunting): authenticate will time out at 770.093212s 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ede59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4ee009ac40 0x7f4ee01d76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.129:60670/0 (socket says 172.21.15.129:60670) 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4ede59c640 1 -- 172.21.15.129:0/4260480007 learned_addr learned my addr 172.21.15.129:0/4260480007 (peer_addr_for_me v1:172.21.15.129:0/0) 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 client.?.objecter ms_handle_connect 0x7f4ee009ac40 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 client.?.objecter ms_handle_connect 0x7f4ee00afe80 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 client.?.objecter ms_handle_connect 0x7f4ee0138f70 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054685206 0 0) 0x7f4ee00b1af0 con 0x7f4ee009ac40 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec0004eb0 con 0x7f4ee009ac40 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594093491 0 0) 0x7f4ee01d8080 con 0x7f4ee00afe80 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): my global_id is 4161 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee00b1af0 con 0x7f4ee00afe80 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413781445 0 0) 0x7f4ee00b02f0 con 0x7f4ee0138f70 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee01d8080 con 0x7f4ee0138f70 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3891250328 0 0) 0x7f4ec0004eb0 con 0x7f4ee009ac40 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ee00b02f0 con 0x7f4ee009ac40 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4ed4001c00 con 0x7f4ee009ac40 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): dump: 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.776 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2978193298 0 0) 0x7f4ee01d8080 con 0x7f4ee0138f70 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ec0004eb0 con 0x7f4ee0138f70 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4ed0001930 con 0x7f4ee0138f70 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 10 monclient(hunting): dump: 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.777 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.781+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1616520134 0 0) 0x7f4ee00b1af0 con 0x7f4ee00afe80 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ee01d8080 con 0x7f4ee00afe80 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4ed80018d0 con 0x7f4ee00afe80 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient(hunting): dump: 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.778 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2383121368 0 0) 0x7f4ee00b02f0 con 0x7f4ee009ac40 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 >> v1:172.21.15.107:6790/0 conn(0x7f4ee0138f70 legacy=0x7f4ee00af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 >> v1:172.21.15.107:6789/0 conn(0x7f4ee00afe80 legacy=0x7f4ee01e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 monclient: found mon.b 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee00b0480 con 0x7f4ee009ac40 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4ee4fa3640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4ee4fa3640 10 monclient: _renew_subs 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4ee4fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee01e36d0 con 0x7f4ee009ac40 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4ee4fa3640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4ee4fa3640 10 monclient: _renew_subs 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4ee4fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee01e3a40 con 0x7f4ee009ac40 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4ee4fa3640 1 librados: init done 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ed4004a90 con 0x7f4ee009ac40 2024-03-29T10:57:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4ed4004c00 con 0x7f4ee009ac40 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4edcd99640 4 set_mon_vals no callback set 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 monclient: dump: 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:epoch 1 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:election_strategy: 1 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:12.781 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4ed4005170 con 0x7f4ee009ac40 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2558655826 0 0) 0x7f4ed4006390 con 0x7f4ee009ac40 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 10 client.4153.objecter ms_dispatch 0x7f4ee0000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.782+0000 7f4eceffd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.783+0000 7f4ee4fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.783+0000 7f4ee4fa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.783+0000 7f4ee4fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.783+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec4005080 con 0x7f4ee009ac40 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.785+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4ed40065a0 con 0x7f4ee009ac40 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.785+0000 7f4eceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.785+0000 7f4eceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:12.807 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:12.826+0000 7fc49e442040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:57:12.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.855+0000 7f4ee4fa3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d8e2a8ab-8a6e-4781-b029-8c6277133458", "id": 7}] 2024-03-29T10:57:12.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.855+0000 7f4ee4fa3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d8e2a8ab-8a6e-4781-b029-8c6277133458", "id": 7}] 2024-03-29T10:57:12.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.855+0000 7f4ee4fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:12.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.855+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/4260480007 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d8e2a8ab-8a6e-4781-b029-8c6277133458", "id": 7} v 0) v1 -- 0x7f4ec40058b0 con 0x7f4ee009ac40 2024-03-29T10:57:12.836 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:12.855+0000 7f1b61413040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:57:12.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.859+0000 7f4eceffd640 1 -- 172.21.15.129:0/4260480007 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d8e2a8ab-8a6e-4781-b029-8c6277133458", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1917148524 0 1371665406) 0x7f4ed40067e0 con 0x7f4ee009ac40 2024-03-29T10:57:12.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.859+0000 7f4eceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d8e2a8ab-8a6e-4781-b029-8c6277133458", "id": 7}] 2024-03-29T10:57:12.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.859+0000 7f4eceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:12.839 INFO:teuthology.orchestra.run.smithi129.stdout:7 2024-03-29T10:57:12.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.859+0000 7f4ee4fa3640 10 librados: watch_flush enter 2024-03-29T10:57:12.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.859+0000 7f4ee4fa3640 10 librados: watch_flush exit 2024-03-29T10:57:12.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.859+0000 7f4ee4fa3640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T10:57:12.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.859+0000 7f4ee4fa3640 10 client.4153.objecter successfully canceled tick 2024-03-29T10:57:12.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.859+0000 7f4ee4fa3640 10 monclient: shutdown 2024-03-29T10:57:12.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.860+0000 7f4ee4fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:12.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.860+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/4260480007 >> v1:172.21.15.129:6789/0 conn(0x7f4ee009ac40 legacy=0x7f4ee01d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.860+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/4260480007 shutdown_connections 2024-03-29T10:57:12.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.860+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/4260480007 >> 172.21.15.129:0/4260480007 conn(0x7f4ee0066b00 msgr2=0x7f4ee013c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:12.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.860+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/4260480007 shutdown_connections 2024-03-29T10:57:12.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.860+0000 7f4ee4fa3640 1 -- 172.21.15.129:0/4260480007 wait complete. 2024-03-29T10:57:12.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-29T10:57:12.860+0000 7f4ee4fa3640 1 librados: shutdown 2024-03-29T10:57:12.850 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:57:12.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:57:12.852 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:57:12.853 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:57:12.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:57:12.854 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:57:12.855 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:57:12.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:57:12.857 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:57:12.857 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:57:12.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:57:12.859 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:57:12.859 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:57:12.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:57:12.891 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:12.910+0000 7f1b61413040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:57:12.892 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:57:12.893 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:57:12.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:57:12.895 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:57:12.895 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:57:12.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:57:12.897 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:57:12.897 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:57:12.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:57:12.899 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:57:12.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:12.931 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:12.951+0000 7fc49e442040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:57:12.997 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:13.016+0000 7fc49e442040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.035+0000 7fa06915d640 10 monclient: get_monmap_and_config 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.035+0000 7fa06915d640 10 monclient: build_initial_monmap 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.035+0000 7fa06915d640 10 monclient: monmap: 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:57:13.036593+0000 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:57:13.036593+0000 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:13.016 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:13.017 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:13.017 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 1 Processor -- start 2024-03-29T10:57:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 1 -- start start 2024-03-29T10:57:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 10 monclient: init 2024-03-29T10:57:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0640ae3c0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0640aae90 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 10 monclient(hunting): picked mon.noname-a con 0x7fa064071dc0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640ac7f0 con 0x7fa064071dc0 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640ab370 con 0x7fa0640ae3c0 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640ab890 con 0x7fa0640aae90 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06915d640 10 monclient(hunting): authenticate will time out at 764.304795s 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06359e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa064071dc0 0x7fa06414a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39140/0 (socket says 172.21.15.107:39140) 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06359e640 1 -- 172.21.15.107:0/2284765148 learned_addr learned my addr 172.21.15.107:0/2284765148 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.036+0000 7fa06259c640 1 --1- 172.21.15.107:0/2284765148 >> v1:172.21.15.107:6790/0 conn(0x7fa0640aae90 0x7fa0640acef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49278/0 (socket says 172.21.15.107:49278) 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702374255 0 0) 0x7fa0640ab890 con 0x7fa0640aae90 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): my global_id is 4175 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa04c003520 con 0x7fa0640aae90 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202762192 0 0) 0x7fa0640ac7f0 con 0x7fa064071dc0 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0640ab890 con 0x7fa064071dc0 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501982506 0 0) 0x7fa0640ab370 con 0x7fa0640ae3c0 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient(hunting): my global_id is 4180 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0640ac7f0 con 0x7fa0640ae3c0 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1553537671 0 0) 0x7fa0640ab890 con 0x7fa064071dc0 2024-03-29T10:57:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 >> v1:172.21.15.107:6790/0 conn(0x7fa0640aae90 legacy=0x7fa0640acef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 >> v1:172.21.15.129:6789/0 conn(0x7fa0640ae3c0 legacy=0x7fa06413c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 monclient: found mon.noname-a 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640ad600 con 0x7fa064071dc0 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: _finish_auth 0 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa06915d640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa05c001920 con 0x7fa064071dc0 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa06915d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: dump: 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa06915d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa05c001a80 con 0x7fa064071dc0 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 1 -- 172.21.15.107:0/2284765148 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa05c001ed0 con 0x7fa064071dc0 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa061d9b640 10 monclient: dump: 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa06915d640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.037+0000 7fa06915d640 4 set_mon_vals no callback set 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.038+0000 7fa06915d640 10 monclient: shutdown 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.038+0000 7fa06915d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.038+0000 7fa06915d640 1 -- 172.21.15.107:0/2284765148 >> v1:172.21.15.107:6789/0 conn(0x7fa064071dc0 legacy=0x7fa06414a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.038+0000 7fa06915d640 1 -- 172.21.15.107:0/2284765148 shutdown_connections 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.038+0000 7fa06915d640 1 -- 172.21.15.107:0/2284765148 >> 172.21.15.107:0/2284765148 conn(0x7fa0640b3750 msgr2=0x7fa0640b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.038+0000 7fa06915d640 1 -- 172.21.15.107:0/2284765148 shutdown_connections 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.038+0000 7fa06915d640 1 -- 172.21.15.107:0/2284765148 wait complete. 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.038+0000 7fa06915d640 10 monclient: build_initial_monmap 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.038+0000 7fa06915d640 10 monclient: monmap: 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 0 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 0.000000 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:created 0.000000 2024-03-29T10:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 1 librados: starting msgr at 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 1 librados: starting objecter 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 1 Processor -- start 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 1 -- start start 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 1 librados: setting wanted keys 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 1 librados: calling monclient init 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 10 monclient: init 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 10 monclient(hunting): picked mon.noname-c con 0x7fa064071dc0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0640aae90 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 10 monclient(hunting): picked mon.noname-a con 0x7fa0640ae3c0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.039+0000 7fa06915d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640ada00 con 0x7fa0640ae3c0 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06406fe40 con 0x7fa0640aae90 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0641332c0 con 0x7fa064071dc0 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 10 monclient(hunting): authenticate will time out at 764.308064s 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa062d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa064071dc0 0x7fa064070550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49288/0 (socket says 172.21.15.107:49288) 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa062d9d640 1 -- 172.21.15.107:0/2659593119 learned_addr learned my addr 172.21.15.107:0/2659593119 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 client.?.objecter ms_handle_connect 0x7fa0640ae3c0 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 client.?.objecter ms_handle_connect 0x7fa064071dc0 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 client.?.objecter ms_handle_connect 0x7fa0640aae90 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498909054 0 0) 0x7fa0640ada00 con 0x7fa0640ae3c0 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa050004eb0 con 0x7fa0640ae3c0 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959354241 0 0) 0x7fa06406fe40 con 0x7fa0640aae90 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0640ada00 con 0x7fa0640aae90 2024-03-29T10:57:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231090331 0 0) 0x7fa0641332c0 con 0x7fa064071dc0 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): my global_id is 4178 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa06406fe40 con 0x7fa064071dc0 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 77390615 0 0) 0x7fa050004eb0 con 0x7fa0640ae3c0 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0641332c0 con 0x7fa0640ae3c0 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa05c001740 con 0x7fa0640ae3c0 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:13.035 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1400540261 0 0) 0x7fa0640ada00 con 0x7fa0640aae90 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa050004eb0 con 0x7fa0640aae90 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa0580012f0 con 0x7fa0640aae90 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:13.036 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 93695613 0 0) 0x7fa06406fe40 con 0x7fa064071dc0 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0640ada00 con 0x7fa064071dc0 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa054001910 con 0x7fa064071dc0 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1986056324 0 0) 0x7fa0641332c0 con 0x7fa0640ae3c0 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 >> v1:172.21.15.107:6790/0 conn(0x7fa064071dc0 legacy=0x7fa064070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 >> v1:172.21.15.129:6789/0 conn(0x7fa0640aae90 legacy=0x7fa0641324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 monclient: found mon.a 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064070c60 con 0x7fa0640ae3c0 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa04b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 5 monclient: authenticate success, global_id 4191 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 10 monclient: _renew_subs 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0641d7430 con 0x7fa0640ae3c0 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 10 monclient: _renew_subs 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:13.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0641d7780 con 0x7fa0640ae3c0 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.040+0000 7fa06915d640 1 librados: init done 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa05c004e10 con 0x7fa0640ae3c0 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa05c005120 con 0x7fa0640ae3c0 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 10 monclient: dump: 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:epoch 1 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:13.040 INFO:teuthology.orchestra.run.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa06159a640 4 set_mon_vals no callback set 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa05c005610 con 0x7fa0640ae3c0 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fa05c006d00 con 0x7fa0640ae3c0 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 10 client.4191.objecter ms_dispatch 0x7fa064000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa04b7fe640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa06915d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa06915d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa06915d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:13.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.041+0000 7fa06915d640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa044005080 con 0x7fa0640ae3c0 2024-03-29T10:57:13.042 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:13.042+0000 7f1b61413040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:57:13.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.043+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa05c006630 con 0x7fa0640ae3c0 2024-03-29T10:57:13.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.043+0000 7fa04b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:13.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.043+0000 7fa04b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:13.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.119+0000 7fa06915d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.119+0000 7fa06915d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.119+0000 7fa06915d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:13.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.119+0000 7fa06915d640 1 -- 172.21.15.107:0/2659593119 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa044005330 con 0x7fa0640ae3c0 2024-03-29T10:57:13.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.120+0000 7fa04b7fe640 1 -- 172.21.15.107:0/2659593119 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fa05c006980 con 0x7fa0640ae3c0 2024-03-29T10:57:13.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.120+0000 7fa04b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.120+0000 7fa04b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:13.101 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-29T10:57:13.101 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.120+0000 7fa06915d640 10 librados: watch_flush enter 2024-03-29T10:57:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 10 librados: watch_flush exit 2024-03-29T10:57:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T10:57:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 10 client.4191.objecter successfully canceled tick 2024-03-29T10:57:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 10 monclient: shutdown 2024-03-29T10:57:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 1 -- 172.21.15.107:0/2659593119 >> v1:172.21.15.107:6789/0 conn(0x7fa0640ae3c0 legacy=0x7fa064132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 1 -- 172.21.15.107:0/2659593119 shutdown_connections 2024-03-29T10:57:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 1 -- 172.21.15.107:0/2659593119 >> 172.21.15.107:0/2659593119 conn(0x7fa0640b3750 msgr2=0x7fa064069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 1 -- 172.21.15.107:0/2659593119 shutdown_connections 2024-03-29T10:57:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 1 -- 172.21.15.107:0/2659593119 wait complete. 2024-03-29T10:57:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-29T10:57:13.121+0000 7fa06915d640 1 librados: shutdown 2024-03-29T10:57:13.108 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:13.128+0000 7fc49e442040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:57:13.111 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:57:13.112 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:57:13.133 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:13.152+0000 7f1b61413040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:57:13.203 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:13.223+0000 7fc49e442040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:57:13.289 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:13.308+0000 7f1b61413040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:57:13.331 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:13.350+0000 7f1b61413040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:57:13.359 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:13.379+0000 7fc49e442040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:57:13.401 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:13.421+0000 7fc49e442040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:57:13.457 INFO:tasks.ceph.osd.1.smithi107.stderr:2024-03-29T10:57:13.476+0000 7f53b2815640 -1 Falling back to public interface 2024-03-29T10:57:13.459 INFO:tasks.ceph.osd.2.smithi107.stderr:2024-03-29T10:57:13.479+0000 7f3dd121a640 -1 Falling back to public interface 2024-03-29T10:57:13.469 INFO:tasks.ceph.osd.0.smithi107.stderr:2024-03-29T10:57:13.488+0000 7f04c2cc8640 -1 Falling back to public interface 2024-03-29T10:57:13.469 INFO:tasks.ceph.osd.3.smithi107.stderr:2024-03-29T10:57:13.488+0000 7f4613c23640 -1 Falling back to public interface 2024-03-29T10:57:13.469 INFO:tasks.ceph.osd.0.smithi107.stderr:2024-03-29T10:57:13.489+0000 7f04c1c77640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6808/0: (98) Address already in use 2024-03-29T10:57:13.486 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:13.506+0000 7f1b61413040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:57:13.495 INFO:tasks.ceph.osd.4.smithi129.stderr:2024-03-29T10:57:13.515+0000 7f6951a8b640 -1 Falling back to public interface 2024-03-29T10:57:13.501 INFO:tasks.ceph.osd.7.smithi129.stderr:2024-03-29T10:57:13.521+0000 7f42186f9640 -1 Falling back to public interface 2024-03-29T10:57:13.502 INFO:tasks.ceph.osd.6.smithi129.stderr:2024-03-29T10:57:13.522+0000 7f07d3c0d640 -1 Falling back to public interface 2024-03-29T10:57:13.521 INFO:tasks.ceph.osd.5.smithi129.stderr:2024-03-29T10:57:13.541+0000 7efc68c66640 -1 Falling back to public interface 2024-03-29T10:57:13.558 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:13.577+0000 7fc49e442040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:57:13.916 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.025+0000 7f9a14104640 10 monclient: get_monmap_and_config 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.025+0000 7f9a14104640 10 monclient: build_initial_monmap 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.025+0000 7f9a14104640 10 monclient: monmap: 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:57:14.025993+0000 2024-03-29T10:57:14.005 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:57:14.025993+0000 2024-03-29T10:57:14.006 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:14.006 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:14.006 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:14.006 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:14.006 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:14.006 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.025+0000 7f9a14104640 1 Processor -- start 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.025+0000 7f9a14104640 1 -- start start 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.025+0000 7f9a14104640 10 monclient: init 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 10 monclient(hunting): picked mon.noname-a con 0x7f9a0c1457f0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 10 monclient(hunting): picked mon.noname-c con 0x7f9a0c13f650 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 10 monclient(hunting): picked mon.noname-b con 0x7f9a0c0a9d70 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c0aaef0 con 0x7f9a0c1457f0 2024-03-29T10:57:14.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c145bd0 con 0x7f9a0c13f650 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c145eb0 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a14104640 10 monclient(hunting): authenticate will time out at 765.294449s 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a11ea0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9a0c1457f0 0x7f9a0c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39156/0 (socket says 172.21.15.107:39156) 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a11ea0640 1 -- 172.21.15.107:0/2894853198 learned_addr learned my addr 172.21.15.107:0/2894853198 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a126a1640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9a0c0a9d70 0x7f9a0c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49304/0 (socket says 172.21.15.107:49304) 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446463166 0 0) 0x7f9a0c145eb0 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 10 monclient(hunting): my global_id is 4181 2024-03-29T10:57:14.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f8003520 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.017 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740273713 0 0) 0x7f9a0c0aaef0 con 0x7f9a0c1457f0 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c145eb0 con 0x7f9a0c1457f0 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1677796196 0 0) 0x7f99f8003520 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 >> v1:172.21.15.129:6789/0 conn(0x7f9a0c13f650 legacy=0x7f9a0c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.026+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 >> v1:172.21.15.107:6789/0 conn(0x7f9a0c1457f0 legacy=0x7f9a0c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: _finish_hunting 0 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 1 monclient: found mon.noname-b 2024-03-29T10:57:14.018 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c146340 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: _finish_auth 0 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: _check_auth_tickets 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9a04001a10 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 5 monclient: authenticate success, global_id 4181 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: dump: 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:14.019 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a04001bf0 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:14.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 1 -- 172.21.15.107:0/2894853198 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9a04002040 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 4 set_mon_vals no callback set 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a10e9e640 10 monclient: dump: 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:14.021 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 10 monclient: shutdown 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 1 -- 172.21.15.107:0/2894853198 >> v1:172.21.15.107:6790/0 conn(0x7f9a0c0a9d70 legacy=0x7f9a0c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 1 -- 172.21.15.107:0/2894853198 shutdown_connections 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 1 -- 172.21.15.107:0/2894853198 >> 172.21.15.107:0/2894853198 conn(0x7f9a0c066a00 msgr2=0x7f9a0c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 1 -- 172.21.15.107:0/2894853198 shutdown_connections 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.027+0000 7f9a14104640 1 -- 172.21.15.107:0/2894853198 wait complete. 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.028+0000 7f9a14104640 10 monclient: build_initial_monmap 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.028+0000 7f9a14104640 10 monclient: monmap: 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:14.022 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.028+0000 7f9a14104640 1 librados: starting msgr at 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.028+0000 7f9a14104640 1 librados: starting objecter 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.028+0000 7f9a14104640 1 Processor -- start 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.028+0000 7f9a14104640 1 -- start start 2024-03-29T10:57:14.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.028+0000 7f9a14104640 1 librados: setting wanted keys 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.028+0000 7f9a14104640 1 librados: calling monclient init 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.028+0000 7f9a14104640 10 monclient: init 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 10 monclient(hunting): picked mon.noname-a con 0x7f9a0c0a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 10 monclient(hunting): picked mon.noname-b con 0x7f9a0c13f650 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 10 monclient(hunting): picked mon.noname-c con 0x7f9a0c1457f0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c1427b0 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c134880 con 0x7f9a0c13f650 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c135bd0 con 0x7f9a0c1457f0 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:14.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a14104640 10 monclient(hunting): authenticate will time out at 765.297308s 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a11ea0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9a0c0a9d70 0x7f9a0c141c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39160/0 (socket says 172.21.15.107:39160) 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a11ea0640 1 -- 172.21.15.107:0/2473479614 learned_addr learned my addr 172.21.15.107:0/2473479614 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 client.?.objecter ms_handle_connect 0x7f9a0c0a9d70 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 client.?.objecter ms_handle_connect 0x7f9a0c1457f0 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 client.?.objecter ms_handle_connect 0x7f9a0c13f650 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232506024 0 0) 0x7f9a0c135bd0 con 0x7f9a0c1457f0 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f0004eb0 con 0x7f9a0c1457f0 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529795619 0 0) 0x7f9a0c1427b0 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): my global_id is 4197 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c135bd0 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676322447 0 0) 0x7f9a0c134880 con 0x7f9a0c13f650 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c1427b0 con 0x7f9a0c13f650 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3503747786 0 0) 0x7f99f0004eb0 con 0x7f9a0c1457f0 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a0c134880 con 0x7f9a0c1457f0 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9a04001f00 con 0x7f9a0c1457f0 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): dump: 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:14.027 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3136272662 0 0) 0x7f9a0c135bd0 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99f0004eb0 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9a08001870 con 0x7f9a0c0a9d70 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.029+0000 7f9a027fc640 10 monclient(hunting): dump: 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:14.029 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3596041487 0 0) 0x7f9a0c1427b0 con 0x7f9a0c13f650 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a0c135bd0 con 0x7f9a0c13f650 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f99fc001870 con 0x7f9a0c13f650 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient(hunting): dump: 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:14.030 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3124138074 0 0) 0x7f9a0c134880 con 0x7f9a0c1457f0 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 >> v1:172.21.15.129:6789/0 conn(0x7f9a0c13f650 legacy=0x7f9a0c133a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 >> v1:172.21.15.107:6789/0 conn(0x7f9a0c0a9d70 legacy=0x7f9a0c141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 monclient: found mon.c 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c135eb0 con 0x7f9a0c1457f0 2024-03-29T10:57:14.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 5 monclient: authenticate success, global_id 4184 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 10 monclient: _renew_subs 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0c142a00 con 0x7f9a0c1457f0 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 10 monclient: _renew_subs 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0c142390 con 0x7f9a0c1457f0 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 1 librados: init done 2024-03-29T10:57:14.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a040020c0 con 0x7f9a0c1457f0 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9a04005050 con 0x7f9a0c1457f0 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a03fff640 4 set_mon_vals no callback set 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 monclient: dump: 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:14.033 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9a04005540 con 0x7f9a0c1457f0 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f9a04006c30 con 0x7f9a0c1457f0 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 10 client.4184.objecter ms_dispatch 0x7f9a0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a027fc640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.030+0000 7f9a14104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.031+0000 7f9a14104640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:14.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.031+0000 7f9a14104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:14.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.031+0000 7f9a14104640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f4005080 con 0x7f9a0c1457f0 2024-03-29T10:57:14.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.033+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9a04006610 con 0x7f9a0c1457f0 2024-03-29T10:57:14.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.033+0000 7f9a027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:14.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.033+0000 7f9a027fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:14.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.094+0000 7f9a14104640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:14.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.095+0000 7f9a14104640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:14.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.095+0000 7f9a14104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:14.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.095+0000 7f9a14104640 1 -- 172.21.15.107:0/2473479614 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99f4005330 con 0x7f9a0c1457f0 2024-03-29T10:57:14.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.095+0000 7f9a027fc640 1 -- 172.21.15.107:0/2473479614 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f9a04006960 con 0x7f9a0c1457f0 2024-03-29T10:57:14.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.095+0000 7f9a027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:14.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.095+0000 7f9a027fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:14.076 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:57:14.076 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 10 librados: watch_flush enter 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 10 librados: watch_flush exit 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 10 client.4184.objecter successfully canceled tick 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 10 monclient: shutdown 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 1 -- 172.21.15.107:0/2473479614 >> v1:172.21.15.107:6790/0 conn(0x7f9a0c1457f0 legacy=0x7f9a0c134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 1 -- 172.21.15.107:0/2473479614 shutdown_connections 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 1 -- 172.21.15.107:0/2473479614 >> 172.21.15.107:0/2473479614 conn(0x7f9a0c066a00 msgr2=0x7f9a0c13ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:14.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.096+0000 7f9a14104640 1 -- 172.21.15.107:0/2473479614 shutdown_connections 2024-03-29T10:57:14.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.097+0000 7f9a14104640 1 -- 172.21.15.107:0/2473479614 wait complete. 2024-03-29T10:57:14.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:14.097+0000 7f9a14104640 1 librados: shutdown 2024-03-29T10:57:14.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:14.301 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:14.320+0000 7f1b61413040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:57:14.380 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:14.399+0000 7fc49e442040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:57:14.578 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:14.597+0000 7f1b61413040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:57:14.630 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:14.650+0000 7f1b61413040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:57:14.653 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:14.672+0000 7fc49e442040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:57:14.704 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:14.724+0000 7fc49e442040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:57:14.745 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:14.765+0000 7f1b61413040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:57:14.823 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-03-29T10:57:14.842+0000 7fc49e442040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:57:15.759 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:15.778+0000 7f1b01d5a640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:57:15.759 INFO:tasks.ceph.mgr.y.smithi107.stderr:2024-03-29T10:57:15.779+0000 7f1b01d5a640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:57:16.946 INFO:tasks.ceph.osd.0.smithi107.stderr:2024-03-29T10:57:16.965+0000 7f04c2cc8640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:57:16.951 INFO:tasks.ceph.osd.4.smithi129.stderr:2024-03-29T10:57:16.970+0000 7f6951a8b640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:57:17.000 INFO:tasks.ceph.osd.1.smithi107.stderr:2024-03-29T10:57:17.017+0000 7f53b2815640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:57:17.006 INFO:tasks.ceph.osd.2.smithi107.stderr:2024-03-29T10:57:17.026+0000 7f3dd121a640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:57:17.010 INFO:tasks.ceph.osd.6.smithi129.stderr:2024-03-29T10:57:17.029+0000 7f07d3c0d640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:57:17.014 INFO:tasks.ceph.osd.3.smithi107.stderr:2024-03-29T10:57:17.033+0000 7f4613c23640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:57:17.016 INFO:tasks.ceph.osd.7.smithi129.stderr:2024-03-29T10:57:17.036+0000 7f42186f9640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:57:17.048 INFO:tasks.ceph.osd.5.smithi129.stderr:2024-03-29T10:57:17.067+0000 7efc68c66640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:57:20.893 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:21.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.071+0000 7fa438435640 10 monclient: get_monmap_and_config 2024-03-29T10:57:21.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.071+0000 7fa438435640 10 monclient: build_initial_monmap 2024-03-29T10:57:21.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.071+0000 7fa438435640 10 monclient: monmap: 2024-03-29T10:57:21.053 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:21.053 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:21.054 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:57:21.073813+0000 2024-03-29T10:57:21.054 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:57:21.073813+0000 2024-03-29T10:57:21.054 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:21.054 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:21.054 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:21.054 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:21.054 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:21.054 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:21.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.072+0000 7fa438435640 1 Processor -- start 2024-03-29T10:57:21.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.073+0000 7fa438435640 1 -- start start 2024-03-29T10:57:21.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.073+0000 7fa438435640 10 monclient: init 2024-03-29T10:57:21.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:21.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:21.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 10 monclient(hunting): picked mon.noname-c con 0x7fa4301457f0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:21.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 10 monclient(hunting): picked mon.noname-b con 0x7fa43013f650 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:21.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 10 monclient(hunting): picked mon.noname-a con 0x7fa4300a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:21.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4359d0640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa43013f650 0x7fa4300b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:52916/0 (socket says 172.21.15.107:52916) 2024-03-29T10:57:21.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4359d0640 1 -- 172.21.15.107:0/1991237742 learned_addr learned my addr 172.21.15.107:0/1991237742 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4300aaef0 con 0x7fa4300a9d70 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 1 -- 172.21.15.107:0/1991237742 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa430145bd0 con 0x7fa4301457f0 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 1 -- 172.21.15.107:0/1991237742 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa430145eb0 con 0x7fa43013f650 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa438435640 10 monclient(hunting): authenticate will time out at 772.343509s 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405808369 0 0) 0x7fa430145bd0 con 0x7fa4301457f0 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 10 monclient(hunting): my global_id is 4231 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa41c003520 con 0x7fa4301457f0 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729689547 0 0) 0x7fa430145eb0 con 0x7fa43013f650 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 10 monclient(hunting): my global_id is 4226 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.074+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa430145bd0 con 0x7fa43013f650 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 814626751 0 0) 0x7fa41c003520 con 0x7fa4301457f0 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 >> v1:172.21.15.107:6790/0 conn(0x7fa43013f650 legacy=0x7fa4300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 >> v1:172.21.15.107:6789/0 conn(0x7fa4300a9d70 legacy=0x7fa43014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: _finish_hunting 0 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 1 monclient: found mon.noname-c 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa430146340 con 0x7fa4301457f0 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: _finish_auth 0 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: _check_auth_tickets 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:21.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa42c0015c0 con 0x7fa4301457f0 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: dump: 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:21.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa438435640 5 monclient: authenticate success, global_id 4231 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa438435640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa42c0017e0 con 0x7fa4301457f0 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 1 -- 172.21.15.107:0/1991237742 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa42c001b70 con 0x7fa4301457f0 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa4351cf640 10 monclient: dump: 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:21.063 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa438435640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.075+0000 7fa438435640 4 set_mon_vals no callback set 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.076+0000 7fa438435640 10 monclient: shutdown 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.076+0000 7fa438435640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.076+0000 7fa438435640 1 -- 172.21.15.107:0/1991237742 >> v1:172.21.15.129:6789/0 conn(0x7fa4301457f0 legacy=0x7fa43013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.076+0000 7fa438435640 1 -- 172.21.15.107:0/1991237742 shutdown_connections 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.076+0000 7fa438435640 1 -- 172.21.15.107:0/1991237742 >> 172.21.15.107:0/1991237742 conn(0x7fa430066a00 msgr2=0x7fa430066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.077+0000 7fa438435640 1 -- 172.21.15.107:0/1991237742 shutdown_connections 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.077+0000 7fa438435640 1 -- 172.21.15.107:0/1991237742 wait complete. 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.077+0000 7fa438435640 10 monclient: build_initial_monmap 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.078+0000 7fa438435640 10 monclient: monmap: 2024-03-29T10:57:21.064 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:21.065 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:21.065 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:57:21.065 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:57:21.065 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:21.065 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:21.065 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:21.065 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:21.065 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:21.065 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.078+0000 7fa438435640 1 librados: starting msgr at 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.078+0000 7fa438435640 1 librados: starting objecter 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.078+0000 7fa438435640 1 Processor -- start 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.078+0000 7fa438435640 1 -- start start 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.078+0000 7fa438435640 1 librados: setting wanted keys 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.078+0000 7fa438435640 1 librados: calling monclient init 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.078+0000 7fa438435640 10 monclient: init 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 10 monclient(hunting): picked mon.noname-b con 0x7fa4300a9d70 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 10 monclient(hunting): picked mon.noname-a con 0x7fa43013f650 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:21.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 10 monclient(hunting): picked mon.noname-c con 0x7fa4301457f0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4300af320 con 0x7fa43013f650 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa4361d1640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa4300a9d70 0x7fa4300b0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:51334/0 (socket says 172.21.15.107:51334) 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa4361d1640 1 -- 172.21.15.107:0/594295957 learned_addr learned my addr 172.21.15.107:0/594295957 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4300acf10 con 0x7fa4300a9d70 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4300ad680 con 0x7fa4301457f0 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa438435640 10 monclient(hunting): authenticate will time out at 772.349566s 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa426ffd640 10 client.?.objecter ms_handle_connect 0x7fa4300a9d70 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.080+0000 7fa426ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 client.?.objecter ms_handle_connect 0x7fa43013f650 2024-03-29T10:57:21.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975944089 0 0) 0x7fa4300acf10 con 0x7fa4300a9d70 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa414004eb0 con 0x7fa4300a9d70 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 client.?.objecter ms_handle_connect 0x7fa4301457f0 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956728220 0 0) 0x7fa4300af320 con 0x7fa43013f650 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.081+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4300acf10 con 0x7fa43013f650 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3257120684 0 0) 0x7fa414004eb0 con 0x7fa4300a9d70 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa414005490 con 0x7fa4300a9d70 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa42c001640 con 0x7fa4300a9d70 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:21.074 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872021414 0 0) 0x7fa4300ad680 con 0x7fa4301457f0 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:21.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): my global_id is 4229 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa414004eb0 con 0x7fa4301457f0 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3582120255 0 0) 0x7fa4300acf10 con 0x7fa43013f650 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4300ad680 con 0x7fa43013f650 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa4200018f0 con 0x7fa43013f650 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.082+0000 7fa426ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:21.076 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3298723269 0 0) 0x7fa414005490 con 0x7fa4300a9d70 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 >> v1:172.21.15.107:6790/0 conn(0x7fa4301457f0 legacy=0x7fa4301d79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 >> v1:172.21.15.107:6789/0 conn(0x7fa43013f650 legacy=0x7fa4301d72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:21.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 1 monclient: found mon.b 2024-03-29T10:57:21.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:21.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:21.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4300ad810 con 0x7fa4300a9d70 2024-03-29T10:57:21.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:21.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:21.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa438435640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa438435640 10 monclient: _renew_subs 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa438435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4300b1600 con 0x7fa4300a9d70 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa438435640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa438435640 10 monclient: _renew_subs 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa438435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4300ae470 con 0x7fa4300a9d70 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa438435640 1 librados: init done 2024-03-29T10:57:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa42c0038a0 con 0x7fa4300a9d70 2024-03-29T10:57:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa42c003c70 con 0x7fa4300a9d70 2024-03-29T10:57:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.083+0000 7fa426ffd640 10 monclient: dump: 2024-03-29T10:57:21.080 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:21.080 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:21.080 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.080 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.084+0000 7fa4349ce640 4 set_mon_vals no callback set 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.085+0000 7fa438435640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.086+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fa42c003dd0 con 0x7fa4300a9d70 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.086+0000 7fa438435640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.086+0000 7fa438435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.086+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa418005080 con 0x7fa4300a9d70 2024-03-29T10:57:21.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.087+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fa42c084ae0 con 0x7fa4300a9d70 2024-03-29T10:57:21.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.087+0000 7fa426ffd640 10 client.4234.objecter ms_dispatch 0x7fa430000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:21.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.087+0000 7fa426ffd640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:21.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.087+0000 7fa426ffd640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:21.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.087+0000 7fa426ffd640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T10:57:21.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.087+0000 7fa4359d0640 10 monclient: get_auth_request con 0x7fa41406e1d0 auth_method 0 2024-03-29T10:57:21.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.087+0000 7fa426ffd640 10 client.4234.objecter ms_handle_connect 0x7fa41406e1d0 2024-03-29T10:57:21.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.094+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa42c0503c0 con 0x7fa4300a9d70 2024-03-29T10:57:21.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.094+0000 7fa426ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:21.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.094+0000 7fa426ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:21.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.538+0000 7fa438435640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:21.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.538+0000 7fa438435640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:21.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.538+0000 7fa438435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:21.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.538+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa418005330 con 0x7fa4300a9d70 2024-03-29T10:57:21.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.539+0000 7fa426ffd640 1 -- 172.21.15.107:0/594295957 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fa42c0522f0 con 0x7fa4300a9d70 2024-03-29T10:57:21.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.539+0000 7fa426ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:21.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.539+0000 7fa426ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:21.522 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:57:21.522 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:21.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.543+0000 7fa438435640 10 librados: watch_flush enter 2024-03-29T10:57:21.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.543+0000 7fa438435640 10 librados: watch_flush exit 2024-03-29T10:57:21.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.543+0000 7fa438435640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T10:57:21.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.543+0000 7fa438435640 10 client.4234.objecter successfully canceled tick 2024-03-29T10:57:21.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.543+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 >> v1:172.21.15.107:6816/882194755 conn(0x7fa41406e1d0 legacy=0x7fa414070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.543+0000 7fa438435640 10 monclient: shutdown 2024-03-29T10:57:21.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.543+0000 7fa438435640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:21.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.543+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 >> v1:172.21.15.129:6789/0 conn(0x7fa4300a9d70 legacy=0x7fa4300b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:21.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.544+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 shutdown_connections 2024-03-29T10:57:21.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.544+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 >> 172.21.15.107:0/594295957 conn(0x7fa430066a00 msgr2=0x7fa43013ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:21.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.544+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 shutdown_connections 2024-03-29T10:57:21.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.544+0000 7fa438435640 1 -- 172.21.15.107:0/594295957 wait complete. 2024-03-29T10:57:21.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:21.544+0000 7fa438435640 1 librados: shutdown 2024-03-29T10:57:21.541 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:28.348 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.524+0000 7f7556a88640 10 monclient: get_monmap_and_config 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.524+0000 7f7556a88640 10 monclient: build_initial_monmap 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.524+0000 7f7556a88640 10 monclient: monmap: 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:57:28.525487+0000 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:57:28.525487+0000 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:28.505 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:28.506 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:28.506 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:28.506 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:28.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.525+0000 7f7556a88640 1 Processor -- start 2024-03-29T10:57:28.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.525+0000 7f7556a88640 1 -- start start 2024-03-29T10:57:28.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.525+0000 7f7556a88640 10 monclient: init 2024-03-29T10:57:28.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.526+0000 7f7556a88640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:28.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.526+0000 7f7556a88640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.526+0000 7f7556a88640 10 monclient(hunting): picked mon.noname-c con 0x7f75500b4330 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.526+0000 7f7556a88640 10 monclient(hunting): picked mon.noname-b con 0x7f75501470b0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f7556a88640 10 monclient(hunting): picked mon.noname-a con 0x7f75501439b0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f7555025640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f75501439b0 0x7f7550151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52766/0 (socket says 172.21.15.107:52766) 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f7555025640 1 -- 172.21.15.107:0/3459473014 learned_addr learned my addr 172.21.15.107:0/3459473014 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f7556a88640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755006f2a0 con 0x7f75501439b0 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f7556a88640 1 -- 172.21.15.107:0/3459473014 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7550146b90 con 0x7f75500b4330 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f7556a88640 1 -- 172.21.15.107:0/3459473014 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75500b2ca0 con 0x7f75501470b0 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f7556a88640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f7556a88640 10 monclient(hunting): authenticate will time out at 779.795282s 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640818392 0 0) 0x7f75500b2ca0 con 0x7f75501470b0 2024-03-29T10:57:28.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 10 monclient(hunting): my global_id is 4232 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7540003520 con 0x7f75501470b0 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514997670 0 0) 0x7f7550146b90 con 0x7f75500b4330 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.527+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75500b2ca0 con 0x7f75500b4330 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3010866072 0 0) 0x7f75500b2ca0 con 0x7f75500b4330 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 >> v1:172.21.15.107:6790/0 conn(0x7f75501470b0 legacy=0x7f75500b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 >> v1:172.21.15.107:6789/0 conn(0x7f75501439b0 legacy=0x7f7550151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 1 monclient: found mon.noname-c 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755006e0c0 con 0x7f75500b4330 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f7556a88640 5 monclient: authenticate success, global_id 4237 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7544001670 con 0x7f75500b4330 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: dump: 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:28.511 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f7556a88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7544001890 con 0x7f75500b4330 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 1 -- 172.21.15.107:0/3459473014 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7544001c20 con 0x7f75500b4330 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f7556a88640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f7556a88640 4 set_mon_vals no callback set 2024-03-29T10:57:28.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.528+0000 7f754f7fe640 10 monclient: dump: 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:28.513 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:28.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.529+0000 7f7556a88640 10 monclient: shutdown 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.529+0000 7f7556a88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.529+0000 7f7556a88640 1 -- 172.21.15.107:0/3459473014 >> v1:172.21.15.129:6789/0 conn(0x7f75500b4330 legacy=0x7f75500b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.529+0000 7f7556a88640 1 -- 172.21.15.107:0/3459473014 shutdown_connections 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.529+0000 7f7556a88640 1 -- 172.21.15.107:0/3459473014 >> 172.21.15.107:0/3459473014 conn(0x7f7550066a00 msgr2=0x7f7550066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.529+0000 7f7556a88640 1 -- 172.21.15.107:0/3459473014 shutdown_connections 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.529+0000 7f7556a88640 1 -- 172.21.15.107:0/3459473014 wait complete. 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.530+0000 7f7556a88640 10 monclient: build_initial_monmap 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.530+0000 7f7556a88640 10 monclient: monmap: 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:28.517 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.530+0000 7f7556a88640 1 librados: starting msgr at 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.530+0000 7f7556a88640 1 librados: starting objecter 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.530+0000 7f7556a88640 1 Processor -- start 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.530+0000 7f7556a88640 1 -- start start 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.530+0000 7f7556a88640 1 librados: setting wanted keys 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.530+0000 7f7556a88640 1 librados: calling monclient init 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.530+0000 7f7556a88640 10 monclient: init 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:28.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 10 monclient(hunting): picked mon.noname-a con 0x7f75500b4330 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 10 monclient(hunting): picked mon.noname-b con 0x7f75501439b0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 10 monclient(hunting): picked mon.noname-c con 0x7f75501470b0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7550121c90 con 0x7f75500b4330 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7550122c10 con 0x7f75501439b0 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7550142df0 con 0x7f75501470b0 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.531+0000 7f7556a88640 10 monclient(hunting): authenticate will time out at 779.799917s 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f7555025640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f75501470b0 0x7f75501df970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55248/0 (socket says 172.21.15.107:55248) 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f7555025640 1 -- 172.21.15.107:0/122050418 learned_addr learned my addr 172.21.15.107:0/122050418 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 client.?.objecter ms_handle_connect 0x7f75501470b0 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 client.?.objecter ms_handle_connect 0x7f75501439b0 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161252180 0 0) 0x7f7550122c10 con 0x7f75501439b0 2024-03-29T10:57:28.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7534004eb0 con 0x7f75501439b0 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 client.?.objecter ms_handle_connect 0x7f75500b4330 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.532+0000 7f754d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.533+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622629250 0 0) 0x7f7550142df0 con 0x7f75501470b0 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.533+0000 7f754d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.533+0000 7f754d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.533+0000 7f754d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.533+0000 7f754d7fa640 10 monclient(hunting): my global_id is 4235 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.533+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7550122c10 con 0x7f75501470b0 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589289068 0 0) 0x7f7550121c90 con 0x7f75500b4330 2024-03-29T10:57:28.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7550142df0 con 0x7f75500b4330 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2574663183 0 0) 0x7f7534004eb0 con 0x7f75501439b0 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7550121c90 con 0x7f75501439b0 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f753c0018d0 con 0x7f75501439b0 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient(hunting): dump: 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:28.521 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3617202372 0 0) 0x7f7550121c90 con 0x7f75501439b0 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 >> v1:172.21.15.107:6790/0 conn(0x7f75501470b0 legacy=0x7f75501df970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 >> v1:172.21.15.107:6789/0 conn(0x7f75500b4330 legacy=0x7f75501426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 monclient: found mon.b 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.534+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7550142f80 con 0x7f75501439b0 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f7556a88640 5 monclient: authenticate success, global_id 4240 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f7556a88640 10 monclient: _renew_subs 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f7556a88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75501431d0 con 0x7f75501439b0 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f7556a88640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f7556a88640 10 monclient: _renew_subs 2024-03-29T10:57:28.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f7556a88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7550120e10 con 0x7f75501439b0 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f7556a88640 1 librados: init done 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f753c003610 con 0x7f75501439b0 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f753c003a20 con 0x7f75501439b0 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754d7fa640 10 monclient: dump: 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:28.524 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:28.525 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:28.525 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:28.525 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:28.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.535+0000 7f754e7fc640 4 set_mon_vals no callback set 2024-03-29T10:57:28.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.536+0000 7f7556a88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:28.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.536+0000 7f7556a88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:28.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.536+0000 7f7556a88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:28.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.536+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7530005080 con 0x7f75501439b0 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.538+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f753c003b80 con 0x7f75501439b0 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.539+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f753c083790 con 0x7f75501439b0 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.539+0000 7f754d7fa640 10 client.4240.objecter ms_dispatch 0x7f7550000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.539+0000 7f754d7fa640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.539+0000 7f754d7fa640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.539+0000 7f754d7fa640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.539+0000 7f7554824640 10 monclient: get_auth_request con 0x7f753406de80 auth_method 0 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.540+0000 7f754d7fa640 10 client.4240.objecter ms_handle_connect 0x7f753406de80 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.545+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f753c04f070 con 0x7f75501439b0 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.545+0000 7f754d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:28.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.545+0000 7f754d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:28.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.856+0000 7f7556a88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:28.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.856+0000 7f7556a88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:28.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.856+0000 7f7556a88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:28.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.856+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7530005330 con 0x7f75501439b0 2024-03-29T10:57:28.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.858+0000 7f754d7fa640 1 -- 172.21.15.107:0/122050418 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f753c050fa0 con 0x7f75501439b0 2024-03-29T10:57:28.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.858+0000 7f754d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:28.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.858+0000 7f754d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:28.838 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:57:28.839 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 10 librados: watch_flush enter 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 10 librados: watch_flush exit 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 10 client.4240.objecter successfully canceled tick 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 >> v1:172.21.15.107:6816/882194755 conn(0x7f753406de80 legacy=0x7f7534070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 10 monclient: shutdown 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 >> v1:172.21.15.129:6789/0 conn(0x7f75501439b0 legacy=0x7f75501df260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 shutdown_connections 2024-03-29T10:57:28.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 >> 172.21.15.107:0/122050418 conn(0x7f7550066a00 msgr2=0x7f7550066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:28.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 shutdown_connections 2024-03-29T10:57:28.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 1 -- 172.21.15.107:0/122050418 wait complete. 2024-03-29T10:57:28.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:28.861+0000 7f7556a88640 1 librados: shutdown 2024-03-29T10:57:28.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:35.662 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:35.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.814+0000 7f2263057640 10 monclient: get_monmap_and_config 2024-03-29T10:57:35.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.814+0000 7f2263057640 10 monclient: build_initial_monmap 2024-03-29T10:57:35.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.814+0000 7f2263057640 10 monclient: monmap: 2024-03-29T10:57:35.795 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:35.795 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:35.795 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:57:35.815330+0000 2024-03-29T10:57:35.795 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:57:35.815330+0000 2024-03-29T10:57:35.796 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:35.796 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:35.796 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:35.796 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:35.796 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:35.796 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.815+0000 7f2263057640 1 Processor -- start 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.815+0000 7f2263057640 1 -- start start 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.815+0000 7f2263057640 10 monclient: init 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 10 monclient(hunting): picked mon.noname-b con 0x7f225c0b4330 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 10 monclient(hunting): picked mon.noname-c con 0x7f225c1470b0 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 10 monclient(hunting): picked mon.noname-a con 0x7f225c1439b0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c06f2a0 con 0x7f225c1439b0 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c146b90 con 0x7f225c1470b0 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c0b2ca0 con 0x7f225c0b4330 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:35.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2263057640 10 monclient(hunting): authenticate will time out at 787.084704s 2024-03-29T10:57:35.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2260df3640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f225c0b4330 0x7f225c0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51874/0 (socket says 172.21.15.107:51874) 2024-03-29T10:57:35.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.816+0000 7f2260df3640 1 -- 172.21.15.107:0/2193560026 learned_addr learned my addr 172.21.15.107:0/2193560026 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:35.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888778541 0 0) 0x7f225c0b2ca0 con 0x7f225c0b4330 2024-03-29T10:57:35.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2240003520 con 0x7f225c0b4330 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244090174 0 0) 0x7f225c146b90 con 0x7f225c1470b0 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f225c0b2ca0 con 0x7f225c1470b0 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165540655 0 0) 0x7f2240003520 con 0x7f225c0b4330 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 >> v1:172.21.15.129:6789/0 conn(0x7f225c1470b0 legacy=0x7f225c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.817+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 >> v1:172.21.15.107:6789/0 conn(0x7f225c1439b0 legacy=0x7f225c151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:35.802 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 1 monclient: found mon.noname-b 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c06e0c0 con 0x7f225c0b4330 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2248001690 con 0x7f225c0b4330 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: dump: 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:35.803 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.804 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.804 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.804 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:35.804 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:35.804 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:35.804 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:35.804 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:35.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f2263057640 5 monclient: authenticate success, global_id 4238 2024-03-29T10:57:35.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f2263057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2248001870 con 0x7f225c0b4330 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 1 -- 172.21.15.107:0/2193560026 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2248001c40 con 0x7f225c0b4330 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f225b7fe640 10 monclient: dump: 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:35.805 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f2263057640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.819+0000 7f2263057640 4 set_mon_vals no callback set 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.820+0000 7f2263057640 10 monclient: shutdown 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.820+0000 7f2263057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.820+0000 7f2263057640 1 -- 172.21.15.107:0/2193560026 >> v1:172.21.15.107:6790/0 conn(0x7f225c0b4330 legacy=0x7f225c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.820+0000 7f2263057640 1 -- 172.21.15.107:0/2193560026 shutdown_connections 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.820+0000 7f2263057640 1 -- 172.21.15.107:0/2193560026 >> 172.21.15.107:0/2193560026 conn(0x7f225c066a00 msgr2=0x7f225c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.821+0000 7f2263057640 1 -- 172.21.15.107:0/2193560026 shutdown_connections 2024-03-29T10:57:35.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.821+0000 7f2263057640 1 -- 172.21.15.107:0/2193560026 wait complete. 2024-03-29T10:57:35.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.822+0000 7f2263057640 10 monclient: build_initial_monmap 2024-03-29T10:57:35.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.822+0000 7f2263057640 10 monclient: monmap: 2024-03-29T10:57:35.811 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:35.811 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:35.811 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:57:35.811 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:57:35.811 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:35.811 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.822+0000 7f2263057640 1 librados: starting msgr at 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.822+0000 7f2263057640 1 librados: starting objecter 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.822+0000 7f2263057640 1 Processor -- start 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.822+0000 7f2263057640 1 -- start start 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.822+0000 7f2263057640 1 librados: setting wanted keys 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.822+0000 7f2263057640 1 librados: calling monclient init 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.822+0000 7f2263057640 10 monclient: init 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:35.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 10 monclient(hunting): picked mon.noname-a con 0x7f225c1439b0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 10 monclient(hunting): picked mon.noname-c con 0x7f225c1470b0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 10 monclient(hunting): picked mon.noname-b con 0x7f225c142d90 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c1df820 con 0x7f225c1439b0 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c1df360 con 0x7f225c142d90 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c1df4f0 con 0x7f225c1470b0 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f2263057640 10 monclient(hunting): authenticate will time out at 787.091467s 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f22615f4640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f225c142d90 0x7f225c1208a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:42628/0 (socket says 172.21.15.107:42628) 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f22615f4640 1 -- 172.21.15.107:0/1955580665 learned_addr learned my addr 172.21.15.107:0/1955580665 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f22597fa640 10 client.?.objecter ms_handle_connect 0x7f225c1470b0 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.823+0000 7f22597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 client.?.objecter ms_handle_connect 0x7f225c142d90 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:35.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533269489 0 0) 0x7f225c1df360 con 0x7f225c142d90 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2244004eb0 con 0x7f225c142d90 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601584676 0 0) 0x7f225c1df4f0 con 0x7f225c1470b0 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f225c1df360 con 0x7f225c1470b0 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2371907275 0 0) 0x7f225c1df360 con 0x7f225c1470b0 2024-03-29T10:57:35.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f225c1df4f0 con 0x7f225c1470b0 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3878570725 0 0) 0x7f2244004eb0 con 0x7f225c142d90 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2244005850 con 0x7f225c142d90 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f224c0018e0 con 0x7f225c142d90 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.824+0000 7f22597fa640 10 monclient(hunting): dump: 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:35.815 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.825+0000 7f22597fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.825+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2254001cb0 con 0x7f225c1470b0 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.825+0000 7f22597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.825+0000 7f22597fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.825+0000 7f22597fa640 10 monclient(hunting): dump: 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:35.816 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.825+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2242836806 0 0) 0x7f2244005850 con 0x7f225c142d90 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.825+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 >> v1:172.21.15.107:6790/0 conn(0x7f225c1470b0 legacy=0x7f225c120190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.825+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 >> v1:172.21.15.107:6789/0 conn(0x7f225c1439b0 legacy=0x7f225c1dec50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.826+0000 7f22597fa640 10 monclient: _finish_hunting 0 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.826+0000 7f22597fa640 1 monclient: found mon.b 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.826+0000 7f22597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.826+0000 7f22597fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.826+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c1e9f60 con 0x7f225c142d90 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f22597fa640 10 monclient: _finish_auth 0 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f22597fa640 10 monclient: _check_auth_tickets 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f22597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f2263057640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f2263057640 10 monclient: _renew_subs 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f2263057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c1e6f30 con 0x7f225c142d90 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f224c0035f0 con 0x7f225c142d90 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f22597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f224c003940 con 0x7f225c142d90 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f22597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f22597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f22597fa640 10 monclient: dump: 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:35.818 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f2263057640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f2263057640 10 monclient: _renew_subs 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f2263057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f225c1e7460 con 0x7f225c142d90 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f2263057640 1 librados: init done 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.827+0000 7f225affd640 4 set_mon_vals no callback set 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.828+0000 7f2263057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.828+0000 7f2263057640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.828+0000 7f2263057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:35.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.828+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225c122690 con 0x7f225c142d90 2024-03-29T10:57:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.830+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f224c003aa0 con 0x7f225c142d90 2024-03-29T10:57:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.830+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f224c084920 con 0x7f225c142d90 2024-03-29T10:57:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.830+0000 7f22597fa640 10 client.4246.objecter ms_dispatch 0x7f225c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.830+0000 7f22597fa640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.830+0000 7f22597fa640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.830+0000 7f22597fa640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-29T10:57:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.830+0000 7f2260df3640 10 monclient: get_auth_request con 0x7f224406e1e0 auth_method 0 2024-03-29T10:57:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.831+0000 7f22597fa640 10 client.4246.objecter ms_handle_connect 0x7f224406e1e0 2024-03-29T10:57:35.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.837+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f224c050200 con 0x7f225c142d90 2024-03-29T10:57:35.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.837+0000 7f22597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:35.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:35.837+0000 7f22597fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:36.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.141+0000 7f2263057640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:36.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.141+0000 7f2263057640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:36.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.141+0000 7f2263057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:57:36.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.141+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f225c075f10 con 0x7f225c142d90 2024-03-29T10:57:36.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.143+0000 7f22597fa640 1 -- 172.21.15.107:0/1955580665 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f224c052130 con 0x7f225c142d90 2024-03-29T10:57:36.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.143+0000 7f22597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:36.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.143+0000 7f22597fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:36.123 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:57:36.124 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:36.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.146+0000 7f2263057640 10 librados: watch_flush enter 2024-03-29T10:57:36.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.146+0000 7f2263057640 10 librados: watch_flush exit 2024-03-29T10:57:36.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.146+0000 7f2263057640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-29T10:57:36.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.146+0000 7f2263057640 10 client.4246.objecter successfully canceled tick 2024-03-29T10:57:36.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.147+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 >> v1:172.21.15.107:6816/882194755 conn(0x7f224406e1e0 legacy=0x7f22440706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.147+0000 7f2263057640 10 monclient: shutdown 2024-03-29T10:57:36.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.147+0000 7f2263057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:36.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.147+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 >> v1:172.21.15.129:6789/0 conn(0x7f225c142d90 legacy=0x7f225c1208a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.148+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 shutdown_connections 2024-03-29T10:57:36.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.148+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 >> 172.21.15.107:0/1955580665 conn(0x7f225c066a00 msgr2=0x7f225c1d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:36.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.148+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 shutdown_connections 2024-03-29T10:57:36.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.148+0000 7f2263057640 1 -- 172.21.15.107:0/1955580665 wait complete. 2024-03-29T10:57:36.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:36.148+0000 7f2263057640 1 librados: shutdown 2024-03-29T10:57:36.143 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:42.949 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:43.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.129+0000 7ff6197c8640 10 monclient: get_monmap_and_config 2024-03-29T10:57:43.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.129+0000 7ff6197c8640 10 monclient: build_initial_monmap 2024-03-29T10:57:43.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.129+0000 7ff6197c8640 10 monclient: monmap: 2024-03-29T10:57:43.110 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:43.111 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:43.111 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:57:43.130876+0000 2024-03-29T10:57:43.111 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:57:43.130876+0000 2024-03-29T10:57:43.111 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:43.111 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:43.111 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:43.111 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:43.111 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:43.111 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:43.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.131+0000 7ff6197c8640 1 Processor -- start 2024-03-29T10:57:43.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.131+0000 7ff6197c8640 1 -- start start 2024-03-29T10:57:43.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.131+0000 7ff6197c8640 10 monclient: init 2024-03-29T10:57:43.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:43.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:43.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 10 monclient(hunting): picked mon.noname-b con 0x7ff6141457f0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:43.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 10 monclient(hunting): picked mon.noname-c con 0x7ff61413f650 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:43.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 10 monclient(hunting): picked mon.noname-a con 0x7ff6140a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:43.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6140aaef0 con 0x7ff6140a9d70 2024-03-29T10:57:43.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff614145bd0 con 0x7ff61413f650 2024-03-29T10:57:43.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff614145eb0 con 0x7ff6141457f0 2024-03-29T10:57:43.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:43.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.132+0000 7ff6197c8640 10 monclient(hunting): authenticate will time out at 794.400236s 2024-03-29T10:57:43.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.135+0000 7ff612ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff6141457f0 0x7ff61413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37727/0 (socket says 172.21.15.107:37727) 2024-03-29T10:57:43.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.135+0000 7ff612ffd640 1 -- 172.21.15.107:0/1151806282 learned_addr learned my addr 172.21.15.107:0/1151806282 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.138+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333959059 0 0) 0x7ff6140aaef0 con 0x7ff6140a9d70 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.138+0000 7ff611ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.138+0000 7ff611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.138+0000 7ff611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.138+0000 7ff611ffb640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.138+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5fc003520 con 0x7ff6140a9d70 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.139+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908669159 0 0) 0x7ff614145eb0 con 0x7ff6141457f0 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.139+0000 7ff611ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.139+0000 7ff611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.139+0000 7ff611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.139+0000 7ff611ffb640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.139+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6140aaef0 con 0x7ff6141457f0 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2370309003 0 0) 0x7ff5fc003520 con 0x7ff6140a9d70 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 >> v1:172.21.15.107:6790/0 conn(0x7ff6141457f0 legacy=0x7ff61413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 >> v1:172.21.15.129:6789/0 conn(0x7ff61413f650 legacy=0x7ff6140b2620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 1 monclient: found mon.noname-a 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614146340 con 0x7ff6140a9d70 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff604001690 con 0x7ff6140a9d70 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: dump: 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6040017f0 con 0x7ff6140a9d70 2024-03-29T10:57:43.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 1 -- 172.21.15.107:0/1151806282 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff604001c00 con 0x7ff6140a9d70 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff611ffb640 10 monclient: dump: 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:43.127 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:43.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff6197c8640 5 monclient: authenticate success, global_id 4254 2024-03-29T10:57:43.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff6197c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:43.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.140+0000 7ff6197c8640 4 set_mon_vals no callback set 2024-03-29T10:57:43.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.141+0000 7ff6197c8640 10 monclient: shutdown 2024-03-29T10:57:43.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.141+0000 7ff6197c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:43.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.141+0000 7ff6197c8640 1 -- 172.21.15.107:0/1151806282 >> v1:172.21.15.107:6789/0 conn(0x7ff6140a9d70 legacy=0x7ff61414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.142+0000 7ff6197c8640 1 -- 172.21.15.107:0/1151806282 shutdown_connections 2024-03-29T10:57:43.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.142+0000 7ff6197c8640 1 -- 172.21.15.107:0/1151806282 >> 172.21.15.107:0/1151806282 conn(0x7ff614066a00 msgr2=0x7ff614066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.144+0000 7ff6197c8640 1 -- 172.21.15.107:0/1151806282 shutdown_connections 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.144+0000 7ff6197c8640 1 -- 172.21.15.107:0/1151806282 wait complete. 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.144+0000 7ff6197c8640 10 monclient: build_initial_monmap 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.145+0000 7ff6197c8640 10 monclient: monmap: 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:43.130 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.145+0000 7ff6197c8640 1 librados: starting msgr at 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.145+0000 7ff6197c8640 1 librados: starting objecter 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.145+0000 7ff6197c8640 1 Processor -- start 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.145+0000 7ff6197c8640 1 -- start start 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.145+0000 7ff6197c8640 1 librados: setting wanted keys 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.146+0000 7ff6197c8640 1 librados: calling monclient init 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.146+0000 7ff6197c8640 10 monclient: init 2024-03-29T10:57:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.146+0000 7ff6197c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:43.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.147+0000 7ff6197c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:43.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.147+0000 7ff6197c8640 10 monclient(hunting): picked mon.noname-c con 0x7ff6140a9d70 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:43.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.147+0000 7ff6197c8640 10 monclient(hunting): picked mon.noname-b con 0x7ff61413f650 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:43.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.147+0000 7ff6197c8640 10 monclient(hunting): picked mon.noname-a con 0x7ff6141457f0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:43.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.147+0000 7ff6197c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6141309a0 con 0x7ff6141457f0 2024-03-29T10:57:43.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.147+0000 7ff6197c8640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff614130720 con 0x7ff61413f650 2024-03-29T10:57:43.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.147+0000 7ff6197c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff614144820 con 0x7ff6140a9d70 2024-03-29T10:57:43.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.147+0000 7ff6197c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:43.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.147+0000 7ff6197c8640 10 monclient(hunting): authenticate will time out at 794.415493s 2024-03-29T10:57:43.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.150+0000 7ff612ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff6140a9d70 0x7ff6141d6460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37793/0 (socket says 172.21.15.107:37793) 2024-03-29T10:57:43.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.150+0000 7ff612ffd640 1 -- 172.21.15.107:0/3280573429 learned_addr learned my addr 172.21.15.107:0/3280573429 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:43.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.151+0000 7ff5fb7fe640 10 client.?.objecter ms_handle_connect 0x7ff6140a9d70 2024-03-29T10:57:43.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.151+0000 7ff5fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:43.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.151+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166412664 0 0) 0x7ff614144820 con 0x7ff6140a9d70 2024-03-29T10:57:43.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.151+0000 7ff5fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:43.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.151+0000 7ff5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:43.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.151+0000 7ff5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:43.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.151+0000 7ff5fb7fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.151+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff600004eb0 con 0x7ff6140a9d70 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.152+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1201046658 0 0) 0x7ff600004eb0 con 0x7ff6140a9d70 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.152+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff614144820 con 0x7ff6140a9d70 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.152+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff608001910 con 0x7ff6140a9d70 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.152+0000 7ff5fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.152+0000 7ff5fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.152+0000 7ff5fb7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:43.134 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:43.135 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:43.135 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:43.135 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.152+0000 7ff5fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.152+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3915412389 0 0) 0x7ff614144820 con 0x7ff6140a9d70 2024-03-29T10:57:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.152+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 >> v1:172.21.15.129:6789/0 conn(0x7ff61413f650 legacy=0x7ff61412f900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:43.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.154+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 >> v1:172.21.15.107:6789/0 conn(0x7ff6141457f0 legacy=0x7ff614130010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.154+0000 7ff5fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:43.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.154+0000 7ff5fb7fe640 1 monclient: found mon.c 2024-03-29T10:57:43.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.154+0000 7ff5fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:43.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.154+0000 7ff5fb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614144b00 con 0x7ff6140a9d70 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff5fb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff5fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff5fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff5fb7fe640 10 client.?.objecter ms_handle_connect 0x7ff61413f650 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff5fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff6197c8640 5 monclient: authenticate success, global_id 4247 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff6197c8640 10 monclient: _renew_subs 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff6197c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff614144de0 con 0x7ff6140a9d70 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff6197c8640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff6197c8640 10 monclient: _renew_subs 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff6197c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:43.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff614135680 con 0x7ff6140a9d70 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.156+0000 7ff6197c8640 1 librados: init done 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.157+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6080047e0 con 0x7ff6140a9d70 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.158+0000 7ff5fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.158+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff608004af0 con 0x7ff6140a9d70 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.158+0000 7ff5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.158+0000 7ff5fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.158+0000 7ff5fb7fe640 10 monclient: dump: 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:43.138 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:43.139 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:43.139 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:43.139 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:43.139 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:43.139 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:43.139 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:43.139 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:43.141 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.159+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7ff60801aac0 con 0x7ff6140a9d70 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.159+0000 7ff6117fa640 4 set_mon_vals no callback set 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.160+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7ff608084b40 con 0x7ff6140a9d70 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.160+0000 7ff5fb7fe640 10 client.4247.objecter ms_dispatch 0x7ff614000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.160+0000 7ff5fb7fe640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.160+0000 7ff5fb7fe640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.160+0000 7ff5fb7fe640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.160+0000 7ff6197c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.160+0000 7ff6197c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.160+0000 7ff6197c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.160+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f4005080 con 0x7ff6140a9d70 2024-03-29T10:57:43.142 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.161+0000 7ff6127fc640 10 monclient: get_auth_request con 0x7ff60006db40 auth_method 0 2024-03-29T10:57:43.143 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.161+0000 7ff5fb7fe640 10 client.4247.objecter ms_handle_connect 0x7ff60006db40 2024-03-29T10:57:43.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.169+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff608050250 con 0x7ff6140a9d70 2024-03-29T10:57:43.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.169+0000 7ff5fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:43.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.169+0000 7ff5fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:43.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.491+0000 7ff6197c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:43.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.491+0000 7ff6197c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:43.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.491+0000 7ff6197c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:43.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.491+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5f4005330 con 0x7ff6140a9d70 2024-03-29T10:57:43.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.495+0000 7ff5fb7fe640 1 -- 172.21.15.107:0/3280573429 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7ff608052350 con 0x7ff6140a9d70 2024-03-29T10:57:43.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.495+0000 7ff5fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:43.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.495+0000 7ff5fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:43.477 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:57:43.477 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:43.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.499+0000 7ff6197c8640 10 librados: watch_flush enter 2024-03-29T10:57:43.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.499+0000 7ff6197c8640 10 librados: watch_flush exit 2024-03-29T10:57:43.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.499+0000 7ff6197c8640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-29T10:57:43.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.499+0000 7ff6197c8640 10 client.4247.objecter successfully canceled tick 2024-03-29T10:57:43.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.499+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 >> v1:172.21.15.107:6816/882194755 conn(0x7ff60006db40 legacy=0x7ff600070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.499+0000 7ff6197c8640 10 monclient: shutdown 2024-03-29T10:57:43.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.499+0000 7ff6197c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:43.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.499+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 >> v1:172.21.15.107:6790/0 conn(0x7ff6140a9d70 legacy=0x7ff6141d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:43.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.500+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 shutdown_connections 2024-03-29T10:57:43.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.500+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 >> 172.21.15.107:0/3280573429 conn(0x7ff614066a00 msgr2=0x7ff61413a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:43.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.503+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 shutdown_connections 2024-03-29T10:57:43.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.504+0000 7ff6197c8640 1 -- 172.21.15.107:0/3280573429 wait complete. 2024-03-29T10:57:43.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:43.505+0000 7ff6197c8640 1 librados: shutdown 2024-03-29T10:57:43.503 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:50.311 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:50.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.486+0000 7fb5851f6640 10 monclient: get_monmap_and_config 2024-03-29T10:57:50.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.486+0000 7fb5851f6640 10 monclient: build_initial_monmap 2024-03-29T10:57:50.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.486+0000 7fb5851f6640 10 monclient: monmap: 2024-03-29T10:57:50.467 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:50.467 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:50.467 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:57:50.487725+0000 2024-03-29T10:57:50.467 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:57:50.487725+0000 2024-03-29T10:57:50.467 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:50.467 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:50.468 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:50.468 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:50.468 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:50.468 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:50.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.490+0000 7fb5851f6640 1 Processor -- start 2024-03-29T10:57:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.492+0000 7fb5851f6640 1 -- start start 2024-03-29T10:57:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.492+0000 7fb5851f6640 10 monclient: init 2024-03-29T10:57:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.493+0000 7fb5851f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.493+0000 7fb5851f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:50.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.494+0000 7fb5851f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5801457f0 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.494+0000 7fb5851f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb58013f650 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.497+0000 7fb57e59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb58013f650 0x7fb5800b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:34489/0 (socket says 172.21.15.107:34489) 2024-03-29T10:57:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.497+0000 7fb57e59c640 1 -- 172.21.15.107:0/3530814757 learned_addr learned my addr 172.21.15.107:0/3530814757 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.497+0000 7fb5851f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb5800a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.497+0000 7fb5851f6640 1 -- 172.21.15.107:0/3530814757 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5800aaef0 con 0x7fb5800a9d70 2024-03-29T10:57:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.497+0000 7fb5851f6640 1 -- 172.21.15.107:0/3530814757 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb580145bd0 con 0x7fb58013f650 2024-03-29T10:57:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.497+0000 7fb5851f6640 1 -- 172.21.15.107:0/3530814757 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb580145eb0 con 0x7fb5801457f0 2024-03-29T10:57:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.497+0000 7fb5851f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:50.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.497+0000 7fb5851f6640 10 monclient(hunting): authenticate will time out at 801.765894s 2024-03-29T10:57:50.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.498+0000 7fb57dd9b640 1 -- 172.21.15.107:0/3530814757 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058323364 0 0) 0x7fb580145bd0 con 0x7fb58013f650 2024-03-29T10:57:50.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.498+0000 7fb57dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.498+0000 7fb57dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.498+0000 7fb57dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.499+0000 7fb57dd9b640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:57:50.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.499+0000 7fb57dd9b640 1 -- 172.21.15.107:0/3530814757 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb568003520 con 0x7fb58013f650 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.500+0000 7fb57dd9b640 1 -- 172.21.15.107:0/3530814757 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3205329588 0 0) 0x7fb568003520 con 0x7fb58013f650 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.500+0000 7fb57dd9b640 1 -- 172.21.15.107:0/3530814757 >> v1:172.21.15.107:6790/0 conn(0x7fb5801457f0 legacy=0x7fb58013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.500+0000 7fb57dd9b640 1 -- 172.21.15.107:0/3530814757 >> v1:172.21.15.107:6789/0 conn(0x7fb5800a9d70 legacy=0x7fb58014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.500+0000 7fb57dd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.500+0000 7fb57dd9b640 1 monclient: found mon.noname-c 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.500+0000 7fb57dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.500+0000 7fb57dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.500+0000 7fb57dd9b640 1 -- 172.21.15.107:0/3530814757 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580146340 con 0x7fb58013f650 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.500+0000 7fb57dd9b640 10 monclient: _finish_auth 0 2024-03-29T10:57:50.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb57dd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb57dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb57dd9b640 1 -- 172.21.15.107:0/3530814757 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb570001690 con 0x7fb58013f650 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb5851f6640 5 monclient: authenticate success, global_id 4252 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb5851f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb57dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb57dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb57dd9b640 10 monclient: dump: 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:50.483 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:50.484 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:50.484 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:50.484 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:50.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb57dd9b640 1 -- 172.21.15.107:0/3530814757 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb570001920 con 0x7fb58013f650 2024-03-29T10:57:50.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb5851f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:50.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb57dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:50.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb57dd9b640 1 -- 172.21.15.107:0/3530814757 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb570001cb0 con 0x7fb58013f650 2024-03-29T10:57:50.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb5851f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:50.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.501+0000 7fb5851f6640 4 set_mon_vals no callback set 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.502+0000 7fb57dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.502+0000 7fb57dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.502+0000 7fb57dd9b640 10 monclient: dump: 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:50.486 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.503+0000 7fb5851f6640 10 monclient: shutdown 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.503+0000 7fb5851f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.503+0000 7fb5851f6640 1 -- 172.21.15.107:0/3530814757 >> v1:172.21.15.129:6789/0 conn(0x7fb58013f650 legacy=0x7fb5800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.503+0000 7fb5851f6640 1 -- 172.21.15.107:0/3530814757 shutdown_connections 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.503+0000 7fb5851f6640 1 -- 172.21.15.107:0/3530814757 >> 172.21.15.107:0/3530814757 conn(0x7fb580066a00 msgr2=0x7fb580066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.504+0000 7fb5851f6640 1 -- 172.21.15.107:0/3530814757 shutdown_connections 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.504+0000 7fb5851f6640 1 -- 172.21.15.107:0/3530814757 wait complete. 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.505+0000 7fb5851f6640 10 monclient: build_initial_monmap 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.505+0000 7fb5851f6640 10 monclient: monmap: 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:50.487 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:50.488 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:50.488 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:50.488 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:50.488 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:50.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.505+0000 7fb5851f6640 1 librados: starting msgr at 2024-03-29T10:57:50.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.505+0000 7fb5851f6640 1 librados: starting objecter 2024-03-29T10:57:50.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.506+0000 7fb5851f6640 1 Processor -- start 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.506+0000 7fb5851f6640 1 -- start start 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.506+0000 7fb5851f6640 1 librados: setting wanted keys 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.506+0000 7fb5851f6640 1 librados: calling monclient init 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.506+0000 7fb5851f6640 10 monclient: init 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.507+0000 7fb5851f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.507+0000 7fb5851f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.507+0000 7fb5851f6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5800a9d70 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.508+0000 7fb5851f6640 10 monclient(hunting): picked mon.noname-c con 0x7fb58013f650 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.508+0000 7fb5851f6640 10 monclient(hunting): picked mon.noname-a con 0x7fb5801457f0 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.508+0000 7fb5851f6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5800ac240 con 0x7fb5801457f0 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.508+0000 7fb5851f6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb580142540 con 0x7fb5800a9d70 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.508+0000 7fb5851f6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb580141c20 con 0x7fb58013f650 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.508+0000 7fb5851f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:50.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.508+0000 7fb5851f6640 10 monclient(hunting): authenticate will time out at 801.776521s 2024-03-29T10:57:50.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb57ed9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb5800a9d70 0x7fb5801d6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:34555/0 (socket says 172.21.15.107:34555) 2024-03-29T10:57:50.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb57ed9d640 1 -- 172.21.15.107:0/1794169123 learned_addr learned my addr 172.21.15.107:0/1794169123 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:50.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb56f7fe640 10 client.?.objecter ms_handle_connect 0x7fb5800a9d70 2024-03-29T10:57:50.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb56f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:50.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773852518 0 0) 0x7fb580142540 con 0x7fb5800a9d70 2024-03-29T10:57:50.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb56f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb56f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb56f7fe640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:57:50.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.512+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb564004eb0 con 0x7fb5800a9d70 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.513+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1846827712 0 0) 0x7fb564004eb0 con 0x7fb5800a9d70 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.513+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb580142540 con 0x7fb5800a9d70 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.513+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb578001b50 con 0x7fb5800a9d70 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.514+0000 7fb56f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.514+0000 7fb56f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.514+0000 7fb56f7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.495 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.514+0000 7fb56f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.514+0000 7fb56f7fe640 10 client.?.objecter ms_handle_connect 0x7fb5801457f0 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.514+0000 7fb56f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 client.?.objecter ms_handle_connect 0x7fb58013f650 2024-03-29T10:57:50.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:50.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429170094 0 0) 0x7fb5800ac240 con 0x7fb5801457f0 2024-03-29T10:57:50.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb564004eb0 con 0x7fb5801457f0 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017527455 0 0) 0x7fb580141c20 con 0x7fb58013f650 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5800ac240 con 0x7fb58013f650 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 566520658 0 0) 0x7fb5800ac240 con 0x7fb58013f650 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb580141c20 con 0x7fb58013f650 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb570001550 con 0x7fb58013f650 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.515+0000 7fb56f7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:50.499 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1284395301 0 0) 0x7fb580141c20 con 0x7fb58013f650 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 >> v1:172.21.15.129:6789/0 conn(0x7fb5800a9d70 legacy=0x7fb5801d6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 >> v1:172.21.15.107:6789/0 conn(0x7fb5801457f0 legacy=0x7fb5801e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 1 monclient: found mon.c 2024-03-29T10:57:50.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:50.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:50.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580142a60 con 0x7fb58013f650 2024-03-29T10:57:50.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:50.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:50.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:50.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb570001730 con 0x7fb58013f650 2024-03-29T10:57:50.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.516+0000 7fb56f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.517+0000 7fb57d59a640 4 set_mon_vals no callback set 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.517+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb570004d90 con 0x7fb58013f650 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.517+0000 7fb56f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.517+0000 7fb56f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.517+0000 7fb56f7fe640 10 monclient: dump: 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:50.502 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.518+0000 7fb5851f6640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.518+0000 7fb5851f6640 10 monclient: _renew_subs 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.518+0000 7fb5851f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.518+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5801d76c0 con 0x7fb58013f650 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.518+0000 7fb5851f6640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.518+0000 7fb5851f6640 10 monclient: _renew_subs 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.518+0000 7fb5851f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.518+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58013add0 con 0x7fb58013f650 2024-03-29T10:57:50.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.518+0000 7fb5851f6640 1 librados: init done 2024-03-29T10:57:50.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.520+0000 7fb5851f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:50.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.520+0000 7fb5851f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:50.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.520+0000 7fb5851f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:50.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.520+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5801426d0 con 0x7fb58013f650 2024-03-29T10:57:50.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.521+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb57001ad60 con 0x7fb58013f650 2024-03-29T10:57:50.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.526+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fb570084940 con 0x7fb58013f650 2024-03-29T10:57:50.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.526+0000 7fb56f7fe640 10 client.4250.objecter ms_dispatch 0x7fb580000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:50.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.526+0000 7fb56f7fe640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:50.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.526+0000 7fb56f7fe640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:50.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.526+0000 7fb56f7fe640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T10:57:50.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.527+0000 7fb57ed9d640 10 monclient: get_auth_request con 0x7fb56406dff0 auth_method 0 2024-03-29T10:57:50.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.527+0000 7fb56f7fe640 10 client.4250.objecter ms_handle_connect 0x7fb56406dff0 2024-03-29T10:57:50.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.533+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb570050190 con 0x7fb58013f650 2024-03-29T10:57:50.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.533+0000 7fb56f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:50.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.533+0000 7fb56f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:50.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.866+0000 7fb5851f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:50.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.867+0000 7fb5851f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:50.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.867+0000 7fb5851f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:57:50.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.867+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5801d7850 con 0x7fb58013f650 2024-03-29T10:57:50.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.868+0000 7fb56f7fe640 1 -- 172.21.15.107:0/1794169123 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fb570001210 con 0x7fb58013f650 2024-03-29T10:57:50.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.868+0000 7fb56f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:50.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.868+0000 7fb56f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:50.849 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:57:50.849 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:50.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 10 librados: watch_flush enter 2024-03-29T10:57:50.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 10 librados: watch_flush exit 2024-03-29T10:57:50.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T10:57:50.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 10 client.4250.objecter successfully canceled tick 2024-03-29T10:57:50.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 >> v1:172.21.15.107:6816/882194755 conn(0x7fb56406dff0 legacy=0x7fb5640704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 10 monclient: shutdown 2024-03-29T10:57:50.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:50.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 >> v1:172.21.15.107:6790/0 conn(0x7fb58013f650 legacy=0x7fb580136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:50.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 shutdown_connections 2024-03-29T10:57:50.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 >> 172.21.15.107:0/1794169123 conn(0x7fb580066a00 msgr2=0x7fb5800b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:50.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 shutdown_connections 2024-03-29T10:57:50.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 1 -- 172.21.15.107:0/1794169123 wait complete. 2024-03-29T10:57:50.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:50.872+0000 7fb5851f6640 1 librados: shutdown 2024-03-29T10:57:50.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:57.676 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:57.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.847+0000 7f75e1be6640 10 monclient: get_monmap_and_config 2024-03-29T10:57:57.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.847+0000 7f75e1be6640 10 monclient: build_initial_monmap 2024-03-29T10:57:57.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.847+0000 7f75e1be6640 10 monclient: monmap: 2024-03-29T10:57:57.829 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:57:57.848013+0000 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:57:57.848013+0000 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:57:57.830 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:57.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.848+0000 7f75e1be6640 1 Processor -- start 2024-03-29T10:57:57.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.848+0000 7f75e1be6640 1 -- start start 2024-03-29T10:57:57.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.848+0000 7f75e1be6640 10 monclient: init 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 10 monclient(hunting): picked mon.noname-c con 0x7f75dc0ae250 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 10 monclient(hunting): picked mon.noname-b con 0x7f75dc13f970 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 10 monclient(hunting): picked mon.noname-a con 0x7f75dc142800 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc0afe00 con 0x7f75dc142800 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc142c10 con 0x7f75dc0ae250 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc0afb20 con 0x7f75dc13f970 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.849+0000 7f75e1be6640 10 monclient(hunting): authenticate will time out at 809.117865s 2024-03-29T10:57:57.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75db7fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f75dc0ae250 0x7f75dc13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:36125/0 (socket says 172.21.15.107:36125) 2024-03-29T10:57:57.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75db7fe640 1 -- 172.21.15.107:0/3967797880 learned_addr learned my addr 172.21.15.107:0/3967797880 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810478859 0 0) 0x7f75dc0afe00 con 0x7f75dc142800 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c4003520 con 0x7f75dc142800 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851157423 0 0) 0x7f75dc142c10 con 0x7f75dc0ae250 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.854+0000 7f75da7fc640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:57:57.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.855+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc0afe00 con 0x7f75dc0ae250 2024-03-29T10:57:57.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2019426698 0 0) 0x7f75dc0afe00 con 0x7f75dc0ae250 2024-03-29T10:57:57.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 >> v1:172.21.15.107:6790/0 conn(0x7f75dc13f970 legacy=0x7f75dc1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:57.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 >> v1:172.21.15.107:6789/0 conn(0x7f75dc142800 legacy=0x7f75dc14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 1 monclient: found mon.noname-c 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc0aab80 con 0x7f75dc0ae250 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f75d40019f0 con 0x7f75dc0ae250 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: dump: 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:57.838 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75d4001c10 con 0x7f75dc0ae250 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 1 -- 172.21.15.107:0/3967797880 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f75d4001fa0 con 0x7f75dc0ae250 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:57.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.857+0000 7f75da7fc640 10 monclient: dump: 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:57.840 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:57.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.858+0000 7f75e1be6640 5 monclient: authenticate success, global_id 4258 2024-03-29T10:57:57.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.858+0000 7f75e1be6640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:57.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.858+0000 7f75e1be6640 4 set_mon_vals no callback set 2024-03-29T10:57:57.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.859+0000 7f75e1be6640 10 monclient: shutdown 2024-03-29T10:57:57.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.859+0000 7f75e1be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:57.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.859+0000 7f75e1be6640 1 -- 172.21.15.107:0/3967797880 >> v1:172.21.15.129:6789/0 conn(0x7f75dc0ae250 legacy=0x7f75dc13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.859+0000 7f75e1be6640 1 -- 172.21.15.107:0/3967797880 shutdown_connections 2024-03-29T10:57:57.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.859+0000 7f75e1be6640 1 -- 172.21.15.107:0/3967797880 >> 172.21.15.107:0/3967797880 conn(0x7f75dc0b35e0 msgr2=0x7f75dc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:57.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.860+0000 7f75e1be6640 1 -- 172.21.15.107:0/3967797880 shutdown_connections 2024-03-29T10:57:57.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.863+0000 7f75e1be6640 1 -- 172.21.15.107:0/3967797880 wait complete. 2024-03-29T10:57:57.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.864+0000 7f75e1be6640 10 monclient: build_initial_monmap 2024-03-29T10:57:57.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.864+0000 7f75e1be6640 10 monclient: monmap: 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.864+0000 7f75e1be6640 1 librados: starting msgr at 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.864+0000 7f75e1be6640 1 librados: starting objecter 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.864+0000 7f75e1be6640 1 Processor -- start 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.864+0000 7f75e1be6640 1 -- start start 2024-03-29T10:57:57.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.864+0000 7f75e1be6640 1 librados: setting wanted keys 2024-03-29T10:57:57.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.864+0000 7f75e1be6640 1 librados: calling monclient init 2024-03-29T10:57:57.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.864+0000 7f75e1be6640 10 monclient: init 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 10 monclient(hunting): picked mon.noname-a con 0x7f75dc0ae250 addr v1:172.21.15.107:6789/0 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 10 monclient(hunting): picked mon.noname-b con 0x7f75dc13f970 addr v1:172.21.15.129:6789/0 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 10 monclient(hunting): picked mon.noname-c con 0x7f75dc142800 addr v1:172.21.15.107:6790/0 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc13bbe0 con 0x7f75dc0ae250 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc143eb0 con 0x7f75dc13f970 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc12b2a0 con 0x7f75dc142800 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.865+0000 7f75e1be6640 10 monclient(hunting): authenticate will time out at 809.133762s 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.869+0000 7f75db7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f75dc0ae250 0x7f75dc1344a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58715/0 (socket says 172.21.15.107:58715) 2024-03-29T10:57:57.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.869+0000 7f75db7fe640 1 -- 172.21.15.107:0/1073201726 learned_addr learned my addr 172.21.15.107:0/1073201726 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.870+0000 7f75c3fff640 10 client.?.objecter ms_handle_connect 0x7f75dc142800 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 client.?.objecter ms_handle_connect 0x7f75dc0ae250 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619042960 0 0) 0x7f75dc12b2a0 con 0x7f75dc142800 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:57:57.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c8004ee0 con 0x7f75dc142800 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791752779 0 0) 0x7f75dc13bbe0 con 0x7f75dc0ae250 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.871+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc12b2a0 con 0x7f75dc0ae250 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2695810395 0 0) 0x7f75dc12b2a0 con 0x7f75dc0ae250 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75dc13bbe0 con 0x7f75dc0ae250 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f75d4001930 con 0x7f75dc0ae250 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 10 monclient(hunting): dump: 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:57.854 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4164544528 0 0) 0x7f75dc13bbe0 con 0x7f75dc0ae250 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 >> v1:172.21.15.107:6790/0 conn(0x7f75dc142800 legacy=0x7f75dc200a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.872+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 >> v1:172.21.15.129:6789/0 conn(0x7f75dc13f970 legacy=0x7f75dc138bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:57.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.874+0000 7f75c3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:57.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.874+0000 7f75c3fff640 1 monclient: found mon.a 2024-03-29T10:57:57.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.875+0000 7f75c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:57.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.875+0000 7f75c3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:57.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.875+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc134bb0 con 0x7f75dc0ae250 2024-03-29T10:57:57.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.875+0000 7f75c3fff640 10 monclient: _finish_auth 0 2024-03-29T10:57:57.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.875+0000 7f75c3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:57.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.875+0000 7f75c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:57.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.877+0000 7f75e1be6640 5 monclient: authenticate success, global_id 4263 2024-03-29T10:57:57.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.877+0000 7f75e1be6640 10 monclient: _renew_subs 2024-03-29T10:57:57.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.877+0000 7f75e1be6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:57.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.877+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc13b200 con 0x7f75dc0ae250 2024-03-29T10:57:57.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.877+0000 7f75e1be6640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:57.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.877+0000 7f75e1be6640 10 monclient: _renew_subs 2024-03-29T10:57:57.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.877+0000 7f75e1be6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:57.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.877+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc13c250 con 0x7f75dc0ae250 2024-03-29T10:57:57.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.877+0000 7f75e1be6640 1 librados: init done 2024-03-29T10:57:57.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.879+0000 7f75e1be6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:57.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.879+0000 7f75e1be6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:57.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.879+0000 7f75e1be6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:57.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.879+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75dc143cf0 con 0x7f75dc0ae250 2024-03-29T10:57:57.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.881+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75d4001a90 con 0x7f75dc0ae250 2024-03-29T10:57:57.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.881+0000 7f75c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:57.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.881+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f75d4004fb0 con 0x7f75dc0ae250 2024-03-29T10:57:57.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.881+0000 7f75c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.881+0000 7f75c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.881+0000 7f75c3fff640 10 monclient: dump: 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:57:57.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.881+0000 7f75d97fa640 4 set_mon_vals no callback set 2024-03-29T10:57:57.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.883+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f75d401af80 con 0x7f75dc0ae250 2024-03-29T10:57:57.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.883+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f75d4084e20 con 0x7f75dc0ae250 2024-03-29T10:57:57.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.883+0000 7f75daffd640 10 monclient: get_auth_request con 0x7f75c806dcf0 auth_method 0 2024-03-29T10:57:57.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.884+0000 7f75c3fff640 10 client.4263.objecter ms_dispatch 0x7f75dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:57.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.884+0000 7f75c3fff640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:57.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.884+0000 7f75c3fff640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:57.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.884+0000 7f75c3fff640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T10:57:57.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.884+0000 7f75c3fff640 10 client.4263.objecter ms_handle_connect 0x7f75c806dcf0 2024-03-29T10:57:57.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.893+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75d4050530 con 0x7f75dc0ae250 2024-03-29T10:57:57.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.893+0000 7f75c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:57.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:57.893+0000 7f75c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:58.195 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.214+0000 7f75e1be6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:58.195 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.214+0000 7f75e1be6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:58.195 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.214+0000 7f75e1be6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:57:58.195 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.214+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75dc075f10 con 0x7f75dc0ae250 2024-03-29T10:57:58.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.216+0000 7f75c3fff640 1 -- 172.21.15.107:0/1073201726 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f75d4052630 con 0x7f75dc0ae250 2024-03-29T10:57:58.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.216+0000 7f75c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:58.196 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.216+0000 7f75c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:58.197 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:57:58.197 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:58.199 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.219+0000 7f75e1be6640 10 librados: watch_flush enter 2024-03-29T10:57:58.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.219+0000 7f75e1be6640 10 librados: watch_flush exit 2024-03-29T10:57:58.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.219+0000 7f75e1be6640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T10:57:58.200 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.219+0000 7f75e1be6640 10 client.4263.objecter successfully canceled tick 2024-03-29T10:57:58.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.221+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 >> v1:172.21.15.107:6816/882194755 conn(0x7f75c806dcf0 legacy=0x7f75c80701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.221+0000 7f75e1be6640 10 monclient: shutdown 2024-03-29T10:57:58.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.221+0000 7f75e1be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:58.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.221+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 >> v1:172.21.15.107:6789/0 conn(0x7f75dc0ae250 legacy=0x7f75dc1344a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:58.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.222+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 shutdown_connections 2024-03-29T10:57:58.202 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.222+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 >> 172.21.15.107:0/1073201726 conn(0x7f75dc0b35e0 msgr2=0x7f75dc0b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:58.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.223+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 shutdown_connections 2024-03-29T10:57:58.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.223+0000 7f75e1be6640 1 -- 172.21.15.107:0/1073201726 wait complete. 2024-03-29T10:57:58.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:57:58.223+0000 7f75e1be6640 1 librados: shutdown 2024-03-29T10:57:58.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:05.029 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:05.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.209+0000 7fcb505c8640 10 monclient: get_monmap_and_config 2024-03-29T10:58:05.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.209+0000 7fcb505c8640 10 monclient: build_initial_monmap 2024-03-29T10:58:05.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.209+0000 7fcb505c8640 10 monclient: monmap: 2024-03-29T10:58:05.190 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:05.190 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:05.190 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:58:05.210751+0000 2024-03-29T10:58:05.190 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:58:05.210751+0000 2024-03-29T10:58:05.191 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:05.191 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:05.191 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:05.191 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:58:05.191 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:58:05.191 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:05.192 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.210+0000 7fcb505c8640 1 Processor -- start 2024-03-29T10:58:05.192 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.210+0000 7fcb505c8640 1 -- start start 2024-03-29T10:58:05.192 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.210+0000 7fcb505c8640 10 monclient: init 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb481422b0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 10 monclient(hunting): picked mon.noname-a con 0x7fcb48142690 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 10 monclient(hunting): picked mon.noname-c con 0x7fcb480b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb48142c80 con 0x7fcb48142690 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480aae40 con 0x7fcb480b17d0 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4813fbd0 con 0x7fcb481422b0 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:05.193 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.211+0000 7fcb505c8640 10 monclient(hunting): authenticate will time out at 816.479718s 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.215+0000 7fcb4e364640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcb481422b0 0x7fcb4813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53983/0 (socket says 172.21.15.107:53983) 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.215+0000 7fcb4db63640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcb48142690 0x7fcb480b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53703/0 (socket says 172.21.15.107:53703) 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.215+0000 7fcb4db63640 1 -- 172.21.15.107:0/4036360027 learned_addr learned my addr 172.21.15.107:0/4036360027 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991554619 0 0) 0x7fcb4813fbd0 con 0x7fcb481422b0 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb34003520 con 0x7fcb481422b0 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743283282 0 0) 0x7fcb48142c80 con 0x7fcb48142690 2024-03-29T10:58:05.203 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4813fbd0 con 0x7fcb48142690 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057703684 0 0) 0x7fcb480aae40 con 0x7fcb480b17d0 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.216+0000 7fcb4d362640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb48142c80 con 0x7fcb480b17d0 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2534814509 0 0) 0x7fcb34003520 con 0x7fcb481422b0 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 >> v1:172.21.15.129:6789/0 conn(0x7fcb480b17d0 legacy=0x7fcb480b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 >> v1:172.21.15.107:6789/0 conn(0x7fcb48142690 legacy=0x7fcb480b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.204 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 10 monclient: _finish_hunting 0 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 1 monclient: found mon.noname-b 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48151940 con 0x7fcb481422b0 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 10 monclient: _finish_auth 0 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 10 monclient: _check_auth_tickets 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb505c8640 5 monclient: authenticate success, global_id 4256 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb505c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcb44001690 con 0x7fcb481422b0 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.217+0000 7fcb4d362640 10 monclient: dump: 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:05.205 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.218+0000 7fcb505c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.218+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb440018b0 con 0x7fcb481422b0 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.219+0000 7fcb4d362640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.219+0000 7fcb505c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.219+0000 7fcb505c8640 4 set_mon_vals no callback set 2024-03-29T10:58:05.206 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.219+0000 7fcb4d362640 1 -- 172.21.15.107:0/4036360027 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcb44001bc0 con 0x7fcb481422b0 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.219+0000 7fcb4d362640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.219+0000 7fcb4d362640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.219+0000 7fcb4d362640 10 monclient: dump: 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:05.207 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.220+0000 7fcb505c8640 10 monclient: shutdown 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.220+0000 7fcb505c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.220+0000 7fcb505c8640 1 -- 172.21.15.107:0/4036360027 >> v1:172.21.15.107:6790/0 conn(0x7fcb481422b0 legacy=0x7fcb4813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.220+0000 7fcb505c8640 1 -- 172.21.15.107:0/4036360027 shutdown_connections 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.220+0000 7fcb505c8640 1 -- 172.21.15.107:0/4036360027 >> 172.21.15.107:0/4036360027 conn(0x7fcb48066a00 msgr2=0x7fcb48066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.220+0000 7fcb505c8640 1 -- 172.21.15.107:0/4036360027 shutdown_connections 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.220+0000 7fcb505c8640 1 -- 172.21.15.107:0/4036360027 wait complete. 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.221+0000 7fcb505c8640 10 monclient: build_initial_monmap 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.221+0000 7fcb505c8640 10 monclient: monmap: 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:05.208 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.221+0000 7fcb505c8640 1 librados: starting msgr at 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.221+0000 7fcb505c8640 1 librados: starting objecter 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.221+0000 7fcb505c8640 1 Processor -- start 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.221+0000 7fcb505c8640 1 -- start start 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.221+0000 7fcb505c8640 1 librados: setting wanted keys 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.221+0000 7fcb505c8640 1 librados: calling monclient init 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.221+0000 7fcb505c8640 10 monclient: init 2024-03-29T10:58:05.209 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:05.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:05.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb480b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:05.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 10 monclient(hunting): picked mon.noname-a con 0x7fcb481422b0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:05.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 10 monclient(hunting): picked mon.noname-c con 0x7fcb48142690 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:05.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4812dc50 con 0x7fcb481422b0 2024-03-29T10:58:05.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4812df30 con 0x7fcb480b17d0 2024-03-29T10:58:05.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb481329b0 con 0x7fcb48142690 2024-03-29T10:58:05.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:05.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.222+0000 7fcb505c8640 10 monclient(hunting): authenticate will time out at 816.490737s 2024-03-29T10:58:05.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.226+0000 7fcb4db63640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcb481422b0 0x7fcb4812e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53769/0 (socket says 172.21.15.107:53769) 2024-03-29T10:58:05.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.226+0000 7fcb4db63640 1 -- 172.21.15.107:0/3546881578 learned_addr learned my addr 172.21.15.107:0/3546881578 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:05.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.226+0000 7fcb3effd640 10 client.?.objecter ms_handle_connect 0x7fcb481422b0 2024-03-29T10:58:05.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.226+0000 7fcb3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:05.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 client.?.objecter ms_handle_connect 0x7fcb48142690 2024-03-29T10:58:05.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:05.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313298528 0 0) 0x7fcb481329b0 con 0x7fcb48142690 2024-03-29T10:58:05.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb2c004eb0 con 0x7fcb48142690 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374738769 0 0) 0x7fcb2c004eb0 con 0x7fcb48142690 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb481329b0 con 0x7fcb48142690 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcb40001b70 con 0x7fcb48142690 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:05.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient(hunting): dump: 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:05.215 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3787525253 0 0) 0x7fcb481329b0 con 0x7fcb48142690 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 >> v1:172.21.15.129:6789/0 conn(0x7fcb480b17d0 legacy=0x7fcb4812bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 >> v1:172.21.15.107:6789/0 conn(0x7fcb481422b0 legacy=0x7fcb4812e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 monclient: found mon.c 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48132b40 con 0x7fcb48142690 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient: _finish_auth 0 2024-03-29T10:58:05.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb505c8640 5 monclient: authenticate success, global_id 4259 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb505c8640 10 monclient: _renew_subs 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb505c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.229+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4812c4a0 con 0x7fcb48142690 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb505c8640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb505c8640 10 monclient: _renew_subs 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb505c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4814e640 con 0x7fcb48142690 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb505c8640 1 librados: init done 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb40003990 con 0x7fcb48142690 2024-03-29T10:58:05.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb3ffff640 4 set_mon_vals no callback set 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcb40003d60 con 0x7fcb48142690 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.230+0000 7fcb3effd640 10 monclient: dump: 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:05.218 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:05.219 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:05.219 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:05.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.232+0000 7fcb505c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:05.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.232+0000 7fcb505c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:05.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.232+0000 7fcb505c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:05.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.232+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb30005080 con 0x7fcb48142690 2024-03-29T10:58:05.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.233+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fcb40003f40 con 0x7fcb48142690 2024-03-29T10:58:05.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.235+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fcb40083cb0 con 0x7fcb48142690 2024-03-29T10:58:05.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.235+0000 7fcb3effd640 10 client.4259.objecter ms_dispatch 0x7fcb48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:05.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.235+0000 7fcb3effd640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:05.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.235+0000 7fcb3effd640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:05.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.235+0000 7fcb3effd640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T10:58:05.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.236+0000 7fcb4e364640 10 monclient: get_auth_request con 0x7fcb2c06dc70 auth_method 0 2024-03-29T10:58:05.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.236+0000 7fcb3effd640 10 client.4259.objecter ms_handle_connect 0x7fcb2c06dc70 2024-03-29T10:58:05.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.241+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb4004f590 con 0x7fcb48142690 2024-03-29T10:58:05.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.241+0000 7fcb3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:05.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.241+0000 7fcb3effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:05.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.564+0000 7fcb505c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:05.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.564+0000 7fcb505c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:05.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.564+0000 7fcb505c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:05.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.564+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb30005330 con 0x7fcb48142690 2024-03-29T10:58:05.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.565+0000 7fcb3effd640 1 -- 172.21.15.107:0/3546881578 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fcb400514c0 con 0x7fcb48142690 2024-03-29T10:58:05.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.565+0000 7fcb3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:05.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.565+0000 7fcb3effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:05.550 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:58:05.550 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:05.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.571+0000 7fcb505c8640 10 librados: watch_flush enter 2024-03-29T10:58:05.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.571+0000 7fcb505c8640 10 librados: watch_flush exit 2024-03-29T10:58:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.571+0000 7fcb505c8640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T10:58:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.571+0000 7fcb505c8640 10 client.4259.objecter successfully canceled tick 2024-03-29T10:58:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.571+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 >> v1:172.21.15.107:6816/882194755 conn(0x7fcb2c06dc70 legacy=0x7fcb2c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.571+0000 7fcb505c8640 10 monclient: shutdown 2024-03-29T10:58:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.571+0000 7fcb505c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.571+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 >> v1:172.21.15.107:6790/0 conn(0x7fcb48142690 legacy=0x7fcb481ef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.573+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 shutdown_connections 2024-03-29T10:58:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.573+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 >> 172.21.15.107:0/3546881578 conn(0x7fcb48066a00 msgr2=0x7fcb4813bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:05.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.573+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 shutdown_connections 2024-03-29T10:58:05.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.573+0000 7fcb505c8640 1 -- 172.21.15.107:0/3546881578 wait complete. 2024-03-29T10:58:05.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:05.574+0000 7fcb505c8640 1 librados: shutdown 2024-03-29T10:58:05.571 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:12.378 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.550+0000 7fb47db20640 10 monclient: get_monmap_and_config 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.550+0000 7fb47db20640 10 monclient: build_initial_monmap 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.550+0000 7fb47db20640 10 monclient: monmap: 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:58:12.551079+0000 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:58:12.551079+0000 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:12.532 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:12.533 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:58:12.533 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:58:12.533 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:12.533 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.551+0000 7fb47db20640 1 Processor -- start 2024-03-29T10:58:12.533 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.551+0000 7fb47db20640 1 -- start start 2024-03-29T10:58:12.533 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.551+0000 7fb47db20640 10 monclient: init 2024-03-29T10:58:12.533 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:12.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:12.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 10 monclient(hunting): picked mon.noname-a con 0x7fb478071f40 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:12.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 10 monclient(hunting): picked mon.noname-b con 0x7fb4780adcb0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:12.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 10 monclient(hunting): picked mon.noname-c con 0x7fb47813f9a0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:12.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4780af1b0 con 0x7fb478071f40 2024-03-29T10:58:12.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4780b0030 con 0x7fb47813f9a0 2024-03-29T10:58:12.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4780aec40 con 0x7fb4780adcb0 2024-03-29T10:58:12.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:12.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.552+0000 7fb47db20640 10 monclient(hunting): authenticate will time out at 823.820558s 2024-03-29T10:58:12.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb477fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb4780adcb0 0x7fb4781457a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49205/0 (socket says 172.21.15.107:49205) 2024-03-29T10:58:12.536 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb477fff640 1 -- 172.21.15.107:0/3886319650 learned_addr learned my addr 172.21.15.107:0/3886319650 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:12.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb4777fe640 1 -- 172.21.15.107:0/3886319650 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464998290 0 0) 0x7fb4780aec40 con 0x7fb4780adcb0 2024-03-29T10:58:12.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb4777fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb4777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb4777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb4777fe640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:58:12.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb4777fe640 1 -- 172.21.15.107:0/3886319650 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb464003520 con 0x7fb4780adcb0 2024-03-29T10:58:12.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb4777fe640 1 -- 172.21.15.107:0/3886319650 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216384972 0 0) 0x7fb464003520 con 0x7fb4780adcb0 2024-03-29T10:58:12.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.556+0000 7fb4777fe640 1 -- 172.21.15.107:0/3886319650 >> v1:172.21.15.129:6789/0 conn(0x7fb47813f9a0 legacy=0x7fb478145eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 1 -- 172.21.15.107:0/3886319650 >> v1:172.21.15.107:6789/0 conn(0x7fb478071f40 legacy=0x7fb47813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 1 monclient: found mon.noname-b 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 1 -- 172.21.15.107:0/3886319650 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4780aedd0 con 0x7fb4780adcb0 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 1 -- 172.21.15.107:0/3886319650 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb468001a10 con 0x7fb4780adcb0 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.542 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb4777fe640 10 monclient: dump: 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb47db20640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.560+0000 7fb47db20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb4777fe640 1 -- 172.21.15.107:0/3886319650 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb468001d20 con 0x7fb4780adcb0 2024-03-29T10:58:12.543 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb4777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb47db20640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb47db20640 4 set_mon_vals no callback set 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb4777fe640 1 -- 172.21.15.107:0/3886319650 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb4680020b0 con 0x7fb4780adcb0 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb4777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb4777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb4777fe640 10 monclient: dump: 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:12.544 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:12.545 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:12.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb47db20640 10 monclient: shutdown 2024-03-29T10:58:12.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb47db20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:12.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.561+0000 7fb47db20640 1 -- 172.21.15.107:0/3886319650 >> v1:172.21.15.107:6790/0 conn(0x7fb4780adcb0 legacy=0x7fb4781457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.562+0000 7fb47db20640 1 -- 172.21.15.107:0/3886319650 shutdown_connections 2024-03-29T10:58:12.545 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.562+0000 7fb47db20640 1 -- 172.21.15.107:0/3886319650 >> 172.21.15.107:0/3886319650 conn(0x7fb4780673e0 msgr2=0x7fb4780b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.562+0000 7fb47db20640 1 -- 172.21.15.107:0/3886319650 shutdown_connections 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.562+0000 7fb47db20640 1 -- 172.21.15.107:0/3886319650 wait complete. 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.563+0000 7fb47db20640 10 monclient: build_initial_monmap 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.563+0000 7fb47db20640 10 monclient: monmap: 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:12.546 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:12.547 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:12.547 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:58:12.547 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:58:12.547 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:12.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.563+0000 7fb47db20640 1 librados: starting msgr at 2024-03-29T10:58:12.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.563+0000 7fb47db20640 1 librados: starting objecter 2024-03-29T10:58:12.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.564+0000 7fb47db20640 1 Processor -- start 2024-03-29T10:58:12.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.564+0000 7fb47db20640 1 -- start start 2024-03-29T10:58:12.547 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.564+0000 7fb47db20640 1 librados: setting wanted keys 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.564+0000 7fb47db20640 1 librados: calling monclient init 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.564+0000 7fb47db20640 10 monclient: init 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 10 monclient(hunting): picked mon.noname-c con 0x7fb4780adcb0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 10 monclient(hunting): picked mon.noname-b con 0x7fb47813f9a0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 10 monclient(hunting): picked mon.noname-a con 0x7fb47813cd20 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47813e7d0 con 0x7fb47813cd20 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47813e4a0 con 0x7fb47813f9a0 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47813dda0 con 0x7fb4780adcb0 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:12.548 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.565+0000 7fb47db20640 10 monclient(hunting): authenticate will time out at 823.833185s 2024-03-29T10:58:12.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb47d31f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb47813cd20 0x7fb4781e6d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59913/0 (socket says 172.21.15.107:59913) 2024-03-29T10:58:12.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb47d31f640 1 -- 172.21.15.107:0/2628251337 learned_addr learned my addr 172.21.15.107:0/2628251337 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:12.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 10 client.?.objecter ms_handle_connect 0x7fb47813cd20 2024-03-29T10:58:12.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:12.551 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 10 client.?.objecter ms_handle_connect 0x7fb4780adcb0 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850856306 0 0) 0x7fb47813e7d0 con 0x7fb47813cd20 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.570+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb458004eb0 con 0x7fb47813cd20 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 client.?.objecter ms_handle_connect 0x7fb47813f9a0 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126354202 0 0) 0x7fb47813dda0 con 0x7fb4780adcb0 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:58:12.552 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb47813e7d0 con 0x7fb4780adcb0 2024-03-29T10:58:12.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536371981 0 0) 0x7fb458004eb0 con 0x7fb47813cd20 2024-03-29T10:58:12.553 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb47813dda0 con 0x7fb47813cd20 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923242947 0 0) 0x7fb47813e4a0 con 0x7fb47813f9a0 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb458004eb0 con 0x7fb47813f9a0 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb460001750 con 0x7fb47813cd20 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): dump: 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:12.554 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3367211207 0 0) 0x7fb47813e7d0 con 0x7fb4780adcb0 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb47813e4a0 con 0x7fb4780adcb0 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb470001510 con 0x7fb4780adcb0 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.571+0000 7fb4757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient(hunting): dump: 2024-03-29T10:58:12.555 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3444385066 0 0) 0x7fb47813dda0 con 0x7fb47813cd20 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 >> v1:172.21.15.107:6790/0 conn(0x7fb4780adcb0 legacy=0x7fb4780ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 >> v1:172.21.15.129:6789/0 conn(0x7fb47813f9a0 legacy=0x7fb478144b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient: _finish_hunting 0 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 1 monclient: found mon.a 2024-03-29T10:58:12.556 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47813e080 con 0x7fb47813cd20 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient: _finish_auth 0 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient: _check_auth_tickets 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb47db20640 5 monclient: authenticate success, global_id 4272 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb47db20640 10 monclient: _renew_subs 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb47db20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb478145230 con 0x7fb47813cd20 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb47db20640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb47db20640 10 monclient: _renew_subs 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb47db20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4780ac730 con 0x7fb47813cd20 2024-03-29T10:58:12.557 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb47db20640 1 librados: init done 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb460001c60 con 0x7fb47813cd20 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb4600049d0 con 0x7fb47813cd20 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb4757fa640 10 monclient: dump: 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:12.558 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:12.559 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:12.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.572+0000 7fb476ffd640 4 set_mon_vals no callback set 2024-03-29T10:58:12.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.575+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb46001a9a0 con 0x7fb47813cd20 2024-03-29T10:58:12.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fb4600848b0 con 0x7fb47813cd20 2024-03-29T10:58:12.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb4757fa640 10 client.4272.objecter ms_dispatch 0x7fb478000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:12.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb4757fa640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:12.559 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb4757fa640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:12.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb4757fa640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T10:58:12.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb47db20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:12.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb47db20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:12.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb47db20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:58:12.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c005080 con 0x7fb47813cd20 2024-03-29T10:58:12.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.576+0000 7fb47cb1e640 10 monclient: get_auth_request con 0x7fb45806df20 auth_method 0 2024-03-29T10:58:12.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.577+0000 7fb4757fa640 10 client.4272.objecter ms_handle_connect 0x7fb45806df20 2024-03-29T10:58:12.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.584+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb460050190 con 0x7fb47813cd20 2024-03-29T10:58:12.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.584+0000 7fb4757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:12.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.584+0000 7fb4757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:12.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.909+0000 7fb47db20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:12.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.909+0000 7fb47db20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:12.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.909+0000 7fb47db20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:58:12.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.909+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb45c005330 con 0x7fb47813cd20 2024-03-29T10:58:12.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.910+0000 7fb4757fa640 1 -- 172.21.15.107:0/2628251337 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fb4600520c0 con 0x7fb47813cd20 2024-03-29T10:58:12.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.910+0000 7fb4757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:12.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.910+0000 7fb4757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:12.893 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:58:12.893 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:12.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.916+0000 7fb47db20640 10 librados: watch_flush enter 2024-03-29T10:58:12.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.916+0000 7fb47db20640 10 librados: watch_flush exit 2024-03-29T10:58:12.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.916+0000 7fb47db20640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T10:58:12.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.916+0000 7fb47db20640 10 client.4272.objecter successfully canceled tick 2024-03-29T10:58:12.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.916+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 >> v1:172.21.15.107:6816/882194755 conn(0x7fb45806df20 legacy=0x7fb4580703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.917+0000 7fb47db20640 10 monclient: shutdown 2024-03-29T10:58:12.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.917+0000 7fb47db20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:12.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.917+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 >> v1:172.21.15.107:6789/0 conn(0x7fb47813cd20 legacy=0x7fb4781e6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:12.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.917+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 shutdown_connections 2024-03-29T10:58:12.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.917+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 >> 172.21.15.107:0/2628251337 conn(0x7fb4780673e0 msgr2=0x7fb478069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:12.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.917+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 shutdown_connections 2024-03-29T10:58:12.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.917+0000 7fb47db20640 1 -- 172.21.15.107:0/2628251337 wait complete. 2024-03-29T10:58:12.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:12.917+0000 7fb47db20640 1 librados: shutdown 2024-03-29T10:58:12.915 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:19.723 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:19.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.891+0000 7f999beaf640 10 monclient: get_monmap_and_config 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.891+0000 7f999beaf640 10 monclient: build_initial_monmap 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.891+0000 7f999beaf640 10 monclient: monmap: 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:58:19.893608+0000 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:58:19.893608+0000 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:58:19.874 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:19.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.892+0000 7f999beaf640 1 Processor -- start 2024-03-29T10:58:19.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 1 -- start start 2024-03-29T10:58:19.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 10 monclient: init 2024-03-29T10:58:19.877 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 10 monclient(hunting): picked mon.noname-c con 0x7f99940a9930 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 10 monclient(hunting): picked mon.noname-a con 0x7f9994149dc0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 10 monclient(hunting): picked mon.noname-b con 0x7f999414ff40 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99940aae60 con 0x7f9994149dc0 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99940b4310 con 0x7f99940a9930 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99940afdd0 con 0x7f999414ff40 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.893+0000 7f999beaf640 10 monclient(hunting): authenticate will time out at 831.162891s 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.894+0000 7f999a44c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f999414ff40 0x7f9994152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35756/0 (socket says 172.21.15.107:35756) 2024-03-29T10:58:19.878 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.894+0000 7f999a44c640 1 -- 172.21.15.107:0/2557208357 learned_addr learned my addr 172.21.15.107:0/2557208357 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.896+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118238665 0 0) 0x7f99940afdd0 con 0x7f999414ff40 2024-03-29T10:58:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.896+0000 7f9998c49640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.896+0000 7f9998c49640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.896+0000 7f9998c49640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.896+0000 7f9998c49640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:58:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.896+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9980003520 con 0x7f999414ff40 2024-03-29T10:58:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.897+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462828008 0 0) 0x7f99940aae60 con 0x7f9994149dc0 2024-03-29T10:58:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.897+0000 7f9998c49640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.897+0000 7f9998c49640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.897+0000 7f9998c49640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.897+0000 7f9998c49640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:58:19.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.897+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99940afdd0 con 0x7f9994149dc0 2024-03-29T10:58:19.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.897+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3308662170 0 0) 0x7f9980003520 con 0x7f999414ff40 2024-03-29T10:58:19.881 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.897+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 >> v1:172.21.15.129:6789/0 conn(0x7f99940a9930 legacy=0x7f9994143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 >> v1:172.21.15.107:6789/0 conn(0x7f9994149dc0 legacy=0x7f999414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 10 monclient: _finish_hunting 0 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 1 monclient: found mon.noname-b 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999414a950 con 0x7f999414ff40 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 10 monclient: _finish_auth 0 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 10 monclient: _check_auth_tickets 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9990001690 con 0x7f999414ff40 2024-03-29T10:58:19.883 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f999beaf640 5 monclient: authenticate success, global_id 4268 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f999beaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f9998c49640 10 monclient: dump: 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:19.884 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.899+0000 7f999beaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.900+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99900018b0 con 0x7f999414ff40 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.900+0000 7f9998c49640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.900+0000 7f999beaf640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.900+0000 7f999beaf640 4 set_mon_vals no callback set 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.900+0000 7f9998c49640 1 -- 172.21.15.107:0/2557208357 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9990001c40 con 0x7f999414ff40 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.900+0000 7f9998c49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.900+0000 7f9998c49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.900+0000 7f9998c49640 10 monclient: dump: 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.885 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:19.886 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:19.886 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:19.886 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:19.886 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:19.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.901+0000 7f999beaf640 10 monclient: shutdown 2024-03-29T10:58:19.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.901+0000 7f999beaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:19.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.901+0000 7f999beaf640 1 -- 172.21.15.107:0/2557208357 >> v1:172.21.15.107:6790/0 conn(0x7f999414ff40 legacy=0x7f9994152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.901+0000 7f999beaf640 1 -- 172.21.15.107:0/2557208357 shutdown_connections 2024-03-29T10:58:19.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.901+0000 7f999beaf640 1 -- 172.21.15.107:0/2557208357 >> 172.21.15.107:0/2557208357 conn(0x7f99940b35e0 msgr2=0x7f99940b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:19.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.903+0000 7f999beaf640 1 -- 172.21.15.107:0/2557208357 shutdown_connections 2024-03-29T10:58:19.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.903+0000 7f999beaf640 1 -- 172.21.15.107:0/2557208357 wait complete. 2024-03-29T10:58:19.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.904+0000 7f999beaf640 10 monclient: build_initial_monmap 2024-03-29T10:58:19.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.904+0000 7f999beaf640 10 monclient: monmap: 2024-03-29T10:58:19.887 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:19.887 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:19.887 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.904+0000 7f999beaf640 1 librados: starting msgr at 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.904+0000 7f999beaf640 1 librados: starting objecter 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.904+0000 7f999beaf640 1 Processor -- start 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.905+0000 7f999beaf640 1 -- start start 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.905+0000 7f999beaf640 1 librados: setting wanted keys 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.905+0000 7f999beaf640 1 librados: calling monclient init 2024-03-29T10:58:19.888 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.905+0000 7f999beaf640 10 monclient: init 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 10 monclient(hunting): picked mon.noname-c con 0x7f9994149dc0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 10 monclient(hunting): picked mon.noname-a con 0x7f999414ff40 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 10 monclient(hunting): picked mon.noname-b con 0x7f9994142f50 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99941e82f0 con 0x7f999414ff40 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99941e8010 con 0x7f9994142f50 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9994144510 con 0x7f9994149dc0 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:19.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f999beaf640 10 monclient(hunting): authenticate will time out at 831.175450s 2024-03-29T10:58:19.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f9999c4b640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9994149dc0 0x7f99941e7900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36538/0 (socket says 172.21.15.107:36538) 2024-03-29T10:58:19.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f9999c4b640 1 -- 172.21.15.107:0/475837923 learned_addr learned my addr 172.21.15.107:0/475837923 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:19.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f998a7fc640 10 client.?.objecter ms_handle_connect 0x7f9994149dc0 2024-03-29T10:58:19.890 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.906+0000 7f998a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652339618 0 0) 0x7f9994144510 con 0x7f9994149dc0 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9978004eb0 con 0x7f9994149dc0 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616129155 0 0) 0x7f9978004eb0 con 0x7f9994149dc0 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9994144510 con 0x7f9994149dc0 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f998c001bf0 con 0x7f9994149dc0 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 10 monclient(hunting): dump: 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:19.891 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 480465055 0 0) 0x7f9994144510 con 0x7f9994149dc0 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 >> v1:172.21.15.129:6789/0 conn(0x7f9994142f50 legacy=0x7f9994145b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:19.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.907+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 >> v1:172.21.15.107:6789/0 conn(0x7f999414ff40 legacy=0x7f99941453f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.910+0000 7f998a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.910+0000 7f998a7fc640 1 monclient: found mon.c 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.910+0000 7f998a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.910+0000 7f998a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.910+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99941f29c0 con 0x7f9994149dc0 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f998a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f998a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f998a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f999beaf640 5 monclient: authenticate success, global_id 4271 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f999beaf640 10 monclient: _renew_subs 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f999beaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99941ef990 con 0x7f9994149dc0 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f998a7fc640 10 client.4271.objecter ms_handle_connect 0x7f999414ff40 2024-03-29T10:58:19.895 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f998a7fc640 10 client.4271.objecter resend_mon_ops 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f999beaf640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f999beaf640 10 monclient: _renew_subs 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f999beaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99941efec0 con 0x7f9994149dc0 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.911+0000 7f999beaf640 1 librados: init done 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.912+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f998c001d50 con 0x7f9994149dc0 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.912+0000 7f998a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.912+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f998c004db0 con 0x7f9994149dc0 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.912+0000 7f998a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.912+0000 7f998a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.912+0000 7f998a7fc640 10 monclient: dump: 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:19.896 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.912+0000 7f998b7fe640 4 set_mon_vals no callback set 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.913+0000 7f999beaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.913+0000 7f999beaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.913+0000 7f999beaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:19.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.913+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997c005080 con 0x7f9994149dc0 2024-03-29T10:58:19.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.914+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f998c01ad80 con 0x7f9994149dc0 2024-03-29T10:58:19.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.915+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f998c084de0 con 0x7f9994149dc0 2024-03-29T10:58:19.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.915+0000 7f998a7fc640 10 client.4271.objecter ms_dispatch 0x7f9994000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:19.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.915+0000 7f998a7fc640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:19.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.915+0000 7f998a7fc640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:19.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.915+0000 7f998a7fc640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T10:58:19.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.915+0000 7f999944a640 10 monclient: get_auth_request con 0x7f997806dc60 auth_method 0 2024-03-29T10:58:19.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.916+0000 7f998a7fc640 10 client.4271.objecter ms_handle_connect 0x7f997806dc60 2024-03-29T10:58:19.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.923+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f998c0b48b0 con 0x7f9994149dc0 2024-03-29T10:58:19.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.923+0000 7f998a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:19.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:19.923+0000 7f998a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:20.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.253+0000 7f999beaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:20.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.253+0000 7f999beaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:20.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.253+0000 7f999beaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:20.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.253+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f997c005330 con 0x7f9994149dc0 2024-03-29T10:58:20.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.254+0000 7f998a7fc640 1 -- 172.21.15.107:0/475837923 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f998c0504f0 con 0x7f9994149dc0 2024-03-29T10:58:20.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.254+0000 7f998a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:20.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.254+0000 7f998a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:20.237 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:58:20.237 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:20.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.258+0000 7f999beaf640 10 librados: watch_flush enter 2024-03-29T10:58:20.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.258+0000 7f999beaf640 10 librados: watch_flush exit 2024-03-29T10:58:20.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.258+0000 7f999beaf640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T10:58:20.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.258+0000 7f999beaf640 10 client.4271.objecter successfully canceled tick 2024-03-29T10:58:20.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.258+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 >> v1:172.21.15.107:6816/882194755 conn(0x7f997806dc60 legacy=0x7f9978070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:20.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.258+0000 7f999beaf640 10 monclient: shutdown 2024-03-29T10:58:20.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.258+0000 7f999beaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:20.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.258+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 >> v1:172.21.15.107:6790/0 conn(0x7f9994149dc0 legacy=0x7f99941e7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:20.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.260+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 shutdown_connections 2024-03-29T10:58:20.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.260+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 >> 172.21.15.107:0/475837923 conn(0x7f99940b35e0 msgr2=0x7f99940b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:20.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.261+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 shutdown_connections 2024-03-29T10:58:20.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.261+0000 7f999beaf640 1 -- 172.21.15.107:0/475837923 wait complete. 2024-03-29T10:58:20.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:20.261+0000 7f999beaf640 1 librados: shutdown 2024-03-29T10:58:20.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:27.068 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:27.213 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.233+0000 7f50d5a25640 10 monclient: get_monmap_and_config 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.233+0000 7f50d5a25640 10 monclient: build_initial_monmap 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.233+0000 7f50d5a25640 10 monclient: monmap: 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:58:27.234426+0000 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:58:27.234426+0000 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:58:27.214 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:27.215 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.234+0000 7f50d5a25640 1 Processor -- start 2024-03-29T10:58:27.215 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.234+0000 7f50d5a25640 1 -- start start 2024-03-29T10:58:27.215 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.234+0000 7f50d5a25640 10 monclient: init 2024-03-29T10:58:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 10 monclient(hunting): picked mon.noname-b con 0x7f50d00708d0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 10 monclient(hunting): picked mon.noname-c con 0x7f50d0143390 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 10 monclient(hunting): picked mon.noname-a con 0x7f50d0149720 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d013dae0 con 0x7f50d0149720 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d0071110 con 0x7f50d0143390 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d00aca90 con 0x7f50d00708d0 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50d5a25640 10 monclient(hunting): authenticate will time out at 838.504196s 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50ceffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f50d00708d0 0x7f50d00b1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45980/0 (socket says 172.21.15.107:45980) 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.236+0000 7f50ceffd640 1 -- 172.21.15.107:0/105034334 learned_addr learned my addr 172.21.15.107:0/105034334 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061709255 0 0) 0x7f50d00aca90 con 0x7f50d00708d0 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:58:27.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b8003520 con 0x7f50d00708d0 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374090826 0 0) 0x7f50d0071110 con 0x7f50d0143390 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d00aca90 con 0x7f50d0143390 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771280756 0 0) 0x7f50d013dae0 con 0x7f50d0149720 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.237+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d0071110 con 0x7f50d0149720 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216724805 0 0) 0x7f50b8003520 con 0x7f50d00708d0 2024-03-29T10:58:27.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 >> v1:172.21.15.129:6789/0 conn(0x7f50d0143390 legacy=0x7f50d01437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 >> v1:172.21.15.107:6789/0 conn(0x7f50d0149720 legacy=0x7f50d014bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 1 monclient: found mon.noname-b 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d0072180 con 0x7f50d00708d0 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f50c4001690 con 0x7f50d00708d0 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: dump: 2024-03-29T10:58:27.226 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50c40017f0 con 0x7f50d00708d0 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 1 -- 172.21.15.107:0/105034334 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f50c4001c40 con 0x7f50d00708d0 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:27.227 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.238+0000 7f50cdffb640 10 monclient: dump: 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.239+0000 7f50d5a25640 5 monclient: authenticate success, global_id 4274 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.239+0000 7f50d5a25640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.239+0000 7f50d5a25640 4 set_mon_vals no callback set 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.240+0000 7f50d5a25640 10 monclient: shutdown 2024-03-29T10:58:27.228 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.240+0000 7f50d5a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.240+0000 7f50d5a25640 1 -- 172.21.15.107:0/105034334 >> v1:172.21.15.107:6790/0 conn(0x7f50d00708d0 legacy=0x7f50d00b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.240+0000 7f50d5a25640 1 -- 172.21.15.107:0/105034334 shutdown_connections 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.240+0000 7f50d5a25640 1 -- 172.21.15.107:0/105034334 >> 172.21.15.107:0/105034334 conn(0x7f50d0067450 msgr2=0x7f50d00b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.241+0000 7f50d5a25640 1 -- 172.21.15.107:0/105034334 shutdown_connections 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.241+0000 7f50d5a25640 1 -- 172.21.15.107:0/105034334 wait complete. 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.242+0000 7f50d5a25640 10 monclient: build_initial_monmap 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.242+0000 7f50d5a25640 10 monclient: monmap: 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:27.229 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:27.230 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:58:27.230 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:58:27.230 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:27.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.243+0000 7f50d5a25640 1 librados: starting msgr at 2024-03-29T10:58:27.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.243+0000 7f50d5a25640 1 librados: starting objecter 2024-03-29T10:58:27.230 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.243+0000 7f50d5a25640 1 Processor -- start 2024-03-29T10:58:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.243+0000 7f50d5a25640 1 -- start start 2024-03-29T10:58:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.244+0000 7f50d5a25640 1 librados: setting wanted keys 2024-03-29T10:58:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.244+0000 7f50d5a25640 1 librados: calling monclient init 2024-03-29T10:58:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.244+0000 7f50d5a25640 10 monclient: init 2024-03-29T10:58:27.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50d5a25640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50d5a25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50d5a25640 10 monclient(hunting): picked mon.noname-b con 0x7f50d00708d0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50ceffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f50d00708d0 0x7f50d0131600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:53898/0 (socket says 172.21.15.107:53898) 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50ceffd640 1 -- 172.21.15.107:0/584974749 learned_addr learned my addr 172.21.15.107:0/584974749 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50d5a25640 10 monclient(hunting): picked mon.noname-a con 0x7f50d0143390 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50b7fff640 10 client.?.objecter ms_handle_connect 0x7f50d00708d0 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50b7fff640 10 client.?.objecter ms_handle_connect 0x7f50d0143390 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.245+0000 7f50b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50d5a25640 10 monclient(hunting): picked mon.noname-c con 0x7f50d0149720 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d00adcc0 con 0x7f50d0143390 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d00adfa0 con 0x7f50d00708d0 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d006cd10 con 0x7f50d0149720 2024-03-29T10:58:27.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50d5a25640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50d5a25640 10 monclient(hunting): authenticate will time out at 838.514516s 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 client.?.objecter ms_handle_connect 0x7f50d0149720 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516163913 0 0) 0x7f50d00adfa0 con 0x7f50d00708d0 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50bc004eb0 con 0x7f50d00708d0 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273211949 0 0) 0x7f50d00adcc0 con 0x7f50d0143390 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.246+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d00adfa0 con 0x7f50d0143390 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972208745 0 0) 0x7f50d006cd10 con 0x7f50d0149720 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d00adcc0 con 0x7f50d0149720 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 790015832 0 0) 0x7f50bc004eb0 con 0x7f50d00708d0 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50d006cd10 con 0x7f50d00708d0 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f50c4001b40 con 0x7f50d00708d0 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:27.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 10 monclient(hunting): dump: 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.247+0000 7f50b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 789283212 0 0) 0x7f50d006cd10 con 0x7f50d00708d0 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 >> v1:172.21.15.107:6790/0 conn(0x7f50d0149720 legacy=0x7f50d01e1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 >> v1:172.21.15.107:6789/0 conn(0x7f50d0143390 legacy=0x7f50d0131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 1 monclient: found mon.b 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d00acd40 con 0x7f50d00708d0 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 10 monclient: _finish_auth 0 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50d5a25640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50d5a25640 10 monclient: _renew_subs 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50d5a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d00aea10 con 0x7f50d00708d0 2024-03-29T10:58:27.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50d5a25640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50d5a25640 10 monclient: _renew_subs 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50d5a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d01e3160 con 0x7f50d00708d0 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50c4001d40 con 0x7f50d00708d0 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50d5a25640 1 librados: init done 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f50c4004dd0 con 0x7f50d00708d0 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50b7fff640 10 monclient: dump: 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:27.241 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.248+0000 7f50cd7fa640 4 set_mon_vals no callback set 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.250+0000 7f50d5a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.250+0000 7f50d5a25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.250+0000 7f50d5a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:27.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.250+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b0005080 con 0x7f50d00708d0 2024-03-29T10:58:27.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.251+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f50c401ada0 con 0x7f50d00708d0 2024-03-29T10:58:27.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.252+0000 7f50ce7fc640 10 monclient: get_auth_request con 0x7f50bc06de30 auth_method 0 2024-03-29T10:58:27.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.252+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f50c4084be0 con 0x7f50d00708d0 2024-03-29T10:58:27.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.252+0000 7f50b7fff640 10 client.4270.objecter ms_dispatch 0x7f50d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:27.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.252+0000 7f50b7fff640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:27.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.252+0000 7f50b7fff640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:27.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.252+0000 7f50b7fff640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T10:58:27.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.252+0000 7f50b7fff640 10 client.4270.objecter ms_handle_connect 0x7f50bc06de30 2024-03-29T10:58:27.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.259+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50c40504c0 con 0x7f50d00708d0 2024-03-29T10:58:27.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.259+0000 7f50b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:27.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.259+0000 7f50b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:27.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.664+0000 7f50d5a25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:27.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.664+0000 7f50d5a25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:27.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.664+0000 7f50d5a25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:27.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.664+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50b0005330 con 0x7f50d00708d0 2024-03-29T10:58:27.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.666+0000 7f50b7fff640 1 -- 172.21.15.107:0/584974749 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f50c40523f0 con 0x7f50d00708d0 2024-03-29T10:58:27.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.666+0000 7f50b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:27.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.666+0000 7f50b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:27.646 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:58:27.646 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:27.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.669+0000 7f50d5a25640 10 librados: watch_flush enter 2024-03-29T10:58:27.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.669+0000 7f50d5a25640 10 librados: watch_flush exit 2024-03-29T10:58:27.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.669+0000 7f50d5a25640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T10:58:27.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.669+0000 7f50d5a25640 10 client.4270.objecter successfully canceled tick 2024-03-29T10:58:27.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.670+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 >> v1:172.21.15.107:6816/882194755 conn(0x7f50bc06de30 legacy=0x7f50bc0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.670+0000 7f50d5a25640 10 monclient: shutdown 2024-03-29T10:58:27.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.670+0000 7f50d5a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:27.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.670+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 >> v1:172.21.15.129:6789/0 conn(0x7f50d00708d0 legacy=0x7f50d0131600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:27.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.670+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 shutdown_connections 2024-03-29T10:58:27.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.670+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 >> 172.21.15.107:0/584974749 conn(0x7f50d0067450 msgr2=0x7f50d00b50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:27.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.670+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 shutdown_connections 2024-03-29T10:58:27.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.670+0000 7f50d5a25640 1 -- 172.21.15.107:0/584974749 wait complete. 2024-03-29T10:58:27.651 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:27.670+0000 7f50d5a25640 1 librados: shutdown 2024-03-29T10:58:27.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:34.474 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:34.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.642+0000 7f6f40d7a640 10 monclient: get_monmap_and_config 2024-03-29T10:58:34.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.642+0000 7f6f40d7a640 10 monclient: build_initial_monmap 2024-03-29T10:58:34.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.642+0000 7f6f40d7a640 10 monclient: monmap: 2024-03-29T10:58:34.628 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:34.628 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:34.628 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:58:34.643055+0000 2024-03-29T10:58:34.628 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:58:34.643055+0000 2024-03-29T10:58:34.628 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.643+0000 7f6f40d7a640 1 Processor -- start 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.643+0000 7f6f40d7a640 1 -- start start 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.643+0000 7f6f40d7a640 10 monclient: init 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.644+0000 7f6f40d7a640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.644+0000 7f6f40d7a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f40d7a640 10 monclient(hunting): picked mon.noname-a con 0x7f6f3c071de0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f40d7a640 10 monclient(hunting): picked mon.noname-c con 0x7f6f3c0721c0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:34.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f40d7a640 10 monclient(hunting): picked mon.noname-b con 0x7f6f3c145ae0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f40d7a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c145650 con 0x7f6f3c071de0 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f40d7a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c0ae260 con 0x7f6f3c0721c0 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f40d7a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c1420d0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f40d7a640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f40d7a640 10 monclient(hunting): authenticate will time out at 845.913249s 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f3ad9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6f3c145ae0 0x7f6f3c145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37030/0 (socket says 172.21.15.107:37030) 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f3ad9d640 1 -- 172.21.15.107:0/1648238706 learned_addr learned my addr 172.21.15.107:0/1648238706 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f3959a640 1 -- 172.21.15.107:0/1648238706 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328601518 0 0) 0x7f6f3c1420d0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f3959a640 10 monclient(hunting): _negotiate 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f3959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:34.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f3959a640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.645+0000 7f6f3959a640 1 -- 172.21.15.107:0/1648238706 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f24003520 con 0x7f6f3c145ae0 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.646+0000 7f6f3959a640 1 -- 172.21.15.107:0/1648238706 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251315655 0 0) 0x7f6f24003520 con 0x7f6f3c145ae0 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.646+0000 7f6f3959a640 1 -- 172.21.15.107:0/1648238706 >> v1:172.21.15.129:6789/0 conn(0x7f6f3c0721c0 legacy=0x7f6f3c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.646+0000 7f6f3959a640 1 -- 172.21.15.107:0/1648238706 >> v1:172.21.15.107:6789/0 conn(0x7f6f3c071de0 legacy=0x7f6f3c0aed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: _finish_hunting 0 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 1 monclient: found mon.noname-b 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 1 -- 172.21.15.107:0/1648238706 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c0b0590 con 0x7f6f3c145ae0 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: _finish_auth 0 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: _check_auth_tickets 2024-03-29T10:58:34.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 1 -- 172.21.15.107:0/1648238706 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6f2c001690 con 0x7f6f3c145ae0 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: dump: 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:34.632 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f40d7a640 5 monclient: authenticate success, global_id 4280 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f40d7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 1 -- 172.21.15.107:0/1648238706 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f2c0017f0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f40d7a640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 1 -- 172.21.15.107:0/1648238706 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6f2c001c40 con 0x7f6f3c145ae0 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f40d7a640 4 set_mon_vals no callback set 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.647+0000 7f6f3959a640 10 monclient: dump: 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:34.633 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:34.634 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.634 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:34.634 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:34.634 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:34.634 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:34.634 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:34.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.648+0000 7f6f40d7a640 10 monclient: shutdown 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.648+0000 7f6f40d7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.648+0000 7f6f40d7a640 1 -- 172.21.15.107:0/1648238706 >> v1:172.21.15.107:6790/0 conn(0x7f6f3c145ae0 legacy=0x7f6f3c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.649+0000 7f6f40d7a640 1 -- 172.21.15.107:0/1648238706 shutdown_connections 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.649+0000 7f6f40d7a640 1 -- 172.21.15.107:0/1648238706 >> 172.21.15.107:0/1648238706 conn(0x7f6f3c06b660 msgr2=0x7f6f3c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.649+0000 7f6f40d7a640 1 -- 172.21.15.107:0/1648238706 shutdown_connections 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.649+0000 7f6f40d7a640 1 -- 172.21.15.107:0/1648238706 wait complete. 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.650+0000 7f6f40d7a640 10 monclient: build_initial_monmap 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.650+0000 7f6f40d7a640 10 monclient: monmap: 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:58:34.640 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.651+0000 7f6f40d7a640 1 librados: starting msgr at 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.651+0000 7f6f40d7a640 1 librados: starting objecter 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.651+0000 7f6f40d7a640 1 Processor -- start 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.651+0000 7f6f40d7a640 1 -- start start 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.651+0000 7f6f40d7a640 1 librados: setting wanted keys 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.651+0000 7f6f40d7a640 1 librados: calling monclient init 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.651+0000 7f6f40d7a640 10 monclient: init 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:34.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 10 monclient(hunting): picked mon.noname-b con 0x7f6f3c071de0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 10 monclient(hunting): picked mon.noname-c con 0x7f6f3c145ae0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f39d9b640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6f3c145ae0 0x7f6f3c13bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38018/0 (socket says 172.21.15.107:38018) 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f3a59c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f6f3c071de0 0x7f6f3c139e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:50048/0 (socket says 172.21.15.107:50048) 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 10 monclient(hunting): picked mon.noname-a con 0x7f6f3c1397b0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c13a740 con 0x7f6f3c1397b0 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c13b200 con 0x7f6f3c071de0 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c13b4e0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.652+0000 7f6f40d7a640 10 monclient(hunting): authenticate will time out at 845.920874s 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f39d9b640 1 -- 172.21.15.107:0/4135826110 learned_addr learned my addr 172.21.15.107:0/4135826110 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 10 client.?.objecter ms_handle_connect 0x7f6f3c145ae0 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:34.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 10 client.?.objecter ms_handle_connect 0x7f6f3c071de0 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996573950 0 0) 0x7f6f3c13b4e0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.653+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f1c004eb0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.654+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559587861 0 0) 0x7f6f3c13b200 con 0x7f6f3c071de0 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.654+0000 7f6f2affd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.654+0000 7f6f2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.654+0000 7f6f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.654+0000 7f6f2affd640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f3c13b4e0 con 0x7f6f3c071de0 2024-03-29T10:58:34.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1973470604 0 0) 0x7f6f1c004eb0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f3c13b200 con 0x7f6f3c145ae0 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6f30001960 con 0x7f6f3c145ae0 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient(hunting): dump: 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:34.644 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 943066949 0 0) 0x7f6f3c13b200 con 0x7f6f3c145ae0 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 >> v1:172.21.15.129:6789/0 conn(0x7f6f3c071de0 legacy=0x7f6f3c139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 >> v1:172.21.15.107:6789/0 conn(0x7f6f3c1397b0 legacy=0x7f6f3c1e5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 monclient: found mon.c 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c13c2f0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient: _finish_auth 0 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:34.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f40d7a640 5 monclient: authenticate success, global_id 4283 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f40d7a640 10 monclient: _renew_subs 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f40d7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f300035f0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f3c070140 con 0x7f6f3c145ae0 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6f30003940 con 0x7f6f3c145ae0 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f2affd640 10 monclient: dump: 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:34.646 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f38d99640 4 set_mon_vals no callback set 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f40d7a640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f40d7a640 10 monclient: _renew_subs 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f40d7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.655+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3c1e78b0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.656+0000 7f6f40d7a640 1 librados: init done 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.657+0000 7f6f40d7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.657+0000 7f6f40d7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:34.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.657+0000 7f6f40d7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.657+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3c075f10 con 0x7f6f3c145ae0 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.657+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f6f30003aa0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.658+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f6f30084a80 con 0x7f6f3c145ae0 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.658+0000 7f6f2affd640 10 client.4283.objecter ms_dispatch 0x7f6f3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.658+0000 7f6f2affd640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.658+0000 7f6f2affd640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.658+0000 7f6f2affd640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.658+0000 7f6f3a59c640 10 monclient: get_auth_request con 0x7f6f1c06dd60 auth_method 0 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.658+0000 7f6f2affd640 10 client.4283.objecter ms_handle_connect 0x7f6f1c06dd60 2024-03-29T10:58:34.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.665+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f30050360 con 0x7f6f3c145ae0 2024-03-29T10:58:34.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.665+0000 7f6f2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:34.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.665+0000 7f6f2affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:34.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.975+0000 7f6f40d7a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.975+0000 7f6f40d7a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.975+0000 7f6f40d7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:34.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.975+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f3c0702d0 con 0x7f6f3c145ae0 2024-03-29T10:58:34.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.976+0000 7f6f2affd640 1 -- 172.21.15.107:0/4135826110 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f6f30052290 con 0x7f6f3c145ae0 2024-03-29T10:58:34.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.976+0000 7f6f2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.976+0000 7f6f2affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:34.956 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:58:34.956 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:34.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 10 librados: watch_flush enter 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 10 librados: watch_flush exit 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 10 client.4283.objecter successfully canceled tick 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 >> v1:172.21.15.107:6816/882194755 conn(0x7f6f1c06dd60 legacy=0x7f6f1c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 10 monclient: shutdown 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 >> v1:172.21.15.107:6790/0 conn(0x7f6f3c145ae0 legacy=0x7f6f3c13bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 shutdown_connections 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 >> 172.21.15.107:0/4135826110 conn(0x7f6f3c06b660 msgr2=0x7f6f3c0b1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 shutdown_connections 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 1 -- 172.21.15.107:0/4135826110 wait complete. 2024-03-29T10:58:34.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:34.979+0000 7f6f40d7a640 1 librados: shutdown 2024-03-29T10:58:34.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:41.781 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:41.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.949+0000 7fb9adeae640 10 monclient: get_monmap_and_config 2024-03-29T10:58:41.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.949+0000 7fb9adeae640 10 monclient: build_initial_monmap 2024-03-29T10:58:41.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.949+0000 7fb9adeae640 10 monclient: monmap: 2024-03-29T10:58:41.929 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:41.929 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:41.929 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:58:41.950093+0000 2024-03-29T10:58:41.930 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:58:41.950093+0000 2024-03-29T10:58:41.930 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:41.930 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:41.930 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:41.930 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:58:41.930 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:58:41.930 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:41.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.950+0000 7fb9adeae640 1 Processor -- start 2024-03-29T10:58:41.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.950+0000 7fb9adeae640 1 -- start start 2024-03-29T10:58:41.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.950+0000 7fb9adeae640 10 monclient: init 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 10 monclient(hunting): picked mon.noname-a con 0x7fb9a81422b0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a8142690 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a80b17d0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a8142c80 con 0x7fb9a81422b0 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a80aae40 con 0x7fb9a8142690 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a813fbd0 con 0x7fb9a80b17d0 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9adeae640 10 monclient(hunting): authenticate will time out at 853.219387s 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9a7fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb9a80b17d0 0x7fb9a80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42472/0 (socket says 172.21.15.107:42472) 2024-03-29T10:58:41.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.951+0000 7fb9a7fff640 1 -- 172.21.15.107:0/3211414835 learned_addr learned my addr 172.21.15.107:0/3211414835 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:41.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.952+0000 7fb9a67fc640 1 -- 172.21.15.107:0/3211414835 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395180231 0 0) 0x7fb9a813fbd0 con 0x7fb9a80b17d0 2024-03-29T10:58:41.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.952+0000 7fb9a67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.952+0000 7fb9a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.952+0000 7fb9a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.952+0000 7fb9a67fc640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:58:41.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.952+0000 7fb9a67fc640 1 -- 172.21.15.107:0/3211414835 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb990003520 con 0x7fb9a80b17d0 2024-03-29T10:58:41.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 1 -- 172.21.15.107:0/3211414835 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3328141200 0 0) 0x7fb990003520 con 0x7fb9a80b17d0 2024-03-29T10:58:41.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 1 -- 172.21.15.107:0/3211414835 >> v1:172.21.15.129:6789/0 conn(0x7fb9a8142690 legacy=0x7fb9a80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:41.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 1 -- 172.21.15.107:0/3211414835 >> v1:172.21.15.107:6789/0 conn(0x7fb9a81422b0 legacy=0x7fb9a813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:41.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 1 monclient: found mon.noname-b 2024-03-29T10:58:41.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 1 -- 172.21.15.107:0/3211414835 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a8151940 con 0x7fb9a80b17d0 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 1 -- 172.21.15.107:0/3211414835 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb998001690 con 0x7fb9a80b17d0 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9a67fc640 10 monclient: dump: 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:41.937 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9adeae640 5 monclient: authenticate success, global_id 4286 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.955+0000 7fb9adeae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9a67fc640 1 -- 172.21.15.107:0/3211414835 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9980018b0 con 0x7fb9a80b17d0 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9adeae640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9adeae640 4 set_mon_vals no callback set 2024-03-29T10:58:41.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9a67fc640 1 -- 172.21.15.107:0/3211414835 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb998001c40 con 0x7fb9a80b17d0 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9a67fc640 10 monclient: dump: 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:41.939 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:41.940 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9adeae640 10 monclient: shutdown 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9adeae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.956+0000 7fb9adeae640 1 -- 172.21.15.107:0/3211414835 >> v1:172.21.15.107:6790/0 conn(0x7fb9a80b17d0 legacy=0x7fb9a80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.957+0000 7fb9adeae640 1 -- 172.21.15.107:0/3211414835 shutdown_connections 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.957+0000 7fb9adeae640 1 -- 172.21.15.107:0/3211414835 >> 172.21.15.107:0/3211414835 conn(0x7fb9a8066a00 msgr2=0x7fb9a8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.957+0000 7fb9adeae640 1 -- 172.21.15.107:0/3211414835 shutdown_connections 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.957+0000 7fb9adeae640 1 -- 172.21.15.107:0/3211414835 wait complete. 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.958+0000 7fb9adeae640 10 monclient: build_initial_monmap 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.958+0000 7fb9adeae640 10 monclient: monmap: 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:41.948 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.958+0000 7fb9adeae640 1 librados: starting msgr at 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.958+0000 7fb9adeae640 1 librados: starting objecter 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.958+0000 7fb9adeae640 1 Processor -- start 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.959+0000 7fb9adeae640 1 -- start start 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.959+0000 7fb9adeae640 1 librados: setting wanted keys 2024-03-29T10:58:41.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.959+0000 7fb9adeae640 1 librados: calling monclient init 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.959+0000 7fb9adeae640 10 monclient: init 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a80b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 10 monclient(hunting): picked mon.noname-a con 0x7fb9a81422b0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a8142690 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a806e870 con 0x7fb9a81422b0 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a81e8610 con 0x7fb9a80b17d0 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a806e010 con 0x7fb9a8142690 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.960+0000 7fb9adeae640 10 monclient(hunting): authenticate will time out at 853.228120s 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.962+0000 7fb9a7fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb9a8142690 0x7fb9a81e7f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43158/0 (socket says 172.21.15.107:43158) 2024-03-29T10:58:41.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.962+0000 7fb9a7fff640 1 -- 172.21.15.107:0/1649277412 learned_addr learned my addr 172.21.15.107:0/1649277412 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.963+0000 7fb98ffff640 10 client.?.objecter ms_handle_connect 0x7fb9a8142690 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.963+0000 7fb98ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.963+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909669695 0 0) 0x7fb9a806e010 con 0x7fb9a8142690 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.963+0000 7fb98ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.963+0000 7fb98ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.963+0000 7fb98ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.963+0000 7fb98ffff640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.963+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb994004eb0 con 0x7fb9a8142690 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.963+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4128308480 0 0) 0x7fb994004eb0 con 0x7fb9a8142690 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9a806e010 con 0x7fb9a8142690 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb998001610 con 0x7fb9a8142690 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:41.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 monclient(hunting): dump: 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 client.?.objecter ms_handle_connect 0x7fb9a80b17d0 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2335304042 0 0) 0x7fb9a806e010 con 0x7fb9a8142690 2024-03-29T10:58:41.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 >> v1:172.21.15.129:6789/0 conn(0x7fb9a80b17d0 legacy=0x7fb9a806f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 >> v1:172.21.15.107:6789/0 conn(0x7fb9a81422b0 legacy=0x7fb9a81e77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 1 monclient: found mon.c 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a806eac0 con 0x7fb9a8142690 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 monclient: _finish_auth 0 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.964+0000 7fb98ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb98ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb9adeae640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb9adeae640 10 monclient: _renew_subs 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb9adeae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a806eda0 con 0x7fb9a8142690 2024-03-29T10:58:41.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb9adeae640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb9adeae640 10 monclient: _renew_subs 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb9adeae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a80b8200 con 0x7fb9a8142690 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb9adeae640 1 librados: init done 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb998004a90 con 0x7fb9a8142690 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb98ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb998004da0 con 0x7fb9a8142690 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb98ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb98ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.965+0000 7fb98ffff640 10 monclient: dump: 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:41.954 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.955 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:41.955 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:41.955 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:41.955 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:41.955 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:41.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.967+0000 7fb9a57fa640 4 set_mon_vals no callback set 2024-03-29T10:58:41.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.967+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb99801ad70 con 0x7fb9a8142690 2024-03-29T10:58:41.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.967+0000 7fb9adeae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.968+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fb998084c30 con 0x7fb9a8142690 2024-03-29T10:58:41.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.968+0000 7fb98ffff640 10 client.4289.objecter ms_dispatch 0x7fb9a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:41.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.968+0000 7fb98ffff640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:41.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.968+0000 7fb98ffff640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:41.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.968+0000 7fb98ffff640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T10:58:41.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.968+0000 7fb9adeae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.970+0000 7fb9adeae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:41.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.970+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb988005080 con 0x7fb9a8142690 2024-03-29T10:58:41.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.970+0000 7fb9a77fe640 10 monclient: get_auth_request con 0x7fb99406dc90 auth_method 0 2024-03-29T10:58:41.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.970+0000 7fb98ffff640 10 client.4289.objecter ms_handle_connect 0x7fb99406dc90 2024-03-29T10:58:41.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.978+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb998050530 con 0x7fb9a8142690 2024-03-29T10:58:41.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.978+0000 7fb98ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:41.978+0000 7fb98ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:42.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.298+0000 7fb9adeae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:42.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.298+0000 7fb9adeae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:42.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.298+0000 7fb9adeae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:58:42.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.298+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb988005330 con 0x7fb9a8142690 2024-03-29T10:58:42.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.300+0000 7fb98ffff640 1 -- 172.21.15.107:0/1649277412 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fb998052460 con 0x7fb9a8142690 2024-03-29T10:58:42.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.300+0000 7fb98ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:42.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.300+0000 7fb98ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:42.281 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:58:42.281 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:42.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.303+0000 7fb9adeae640 10 librados: watch_flush enter 2024-03-29T10:58:42.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.304+0000 7fb9adeae640 10 librados: watch_flush exit 2024-03-29T10:58:42.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.304+0000 7fb9adeae640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T10:58:42.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.304+0000 7fb9adeae640 10 client.4289.objecter successfully canceled tick 2024-03-29T10:58:42.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.306+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 >> v1:172.21.15.107:6816/882194755 conn(0x7fb99406dc90 legacy=0x7fb994070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:42.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.306+0000 7fb9adeae640 10 monclient: shutdown 2024-03-29T10:58:42.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.306+0000 7fb9adeae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:42.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.306+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 >> v1:172.21.15.107:6790/0 conn(0x7fb9a8142690 legacy=0x7fb9a81e7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:42.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.306+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 shutdown_connections 2024-03-29T10:58:42.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.306+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 >> 172.21.15.107:0/1649277412 conn(0x7fb9a8066a00 msgr2=0x7fb9a813e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:42.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.306+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 shutdown_connections 2024-03-29T10:58:42.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.307+0000 7fb9adeae640 1 -- 172.21.15.107:0/1649277412 wait complete. 2024-03-29T10:58:42.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:42.307+0000 7fb9adeae640 1 librados: shutdown 2024-03-29T10:58:42.305 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:49.112 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:49.261 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.280+0000 7fa93d94d640 10 monclient: get_monmap_and_config 2024-03-29T10:58:49.261 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.280+0000 7fa93d94d640 10 monclient: build_initial_monmap 2024-03-29T10:58:49.261 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.280+0000 7fa93d94d640 10 monclient: monmap: 2024-03-29T10:58:49.261 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:49.261 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:49.262 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:58:49.281899+0000 2024-03-29T10:58:49.262 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:58:49.281899+0000 2024-03-29T10:58:49.262 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:49.262 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:49.262 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:49.262 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:58:49.262 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:58:49.262 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:49.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.285+0000 7fa93d94d640 1 Processor -- start 2024-03-29T10:58:49.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.285+0000 7fa93d94d640 1 -- start start 2024-03-29T10:58:49.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.285+0000 7fa93d94d640 10 monclient: init 2024-03-29T10:58:49.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:49.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:49.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 10 monclient(hunting): picked mon.noname-a con 0x7fa93806d9d0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:49.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 10 monclient(hunting): picked mon.noname-b con 0x7fa93813ef30 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:49.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9380b07d0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:49.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9380aed10 con 0x7fa93806d9d0 2024-03-29T10:58:49.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9380b0c40 con 0x7fa9380b07d0 2024-03-29T10:58:49.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9380ab1b0 con 0x7fa93813ef30 2024-03-29T10:58:49.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:49.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.286+0000 7fa93d94d640 10 monclient(hunting): authenticate will time out at 860.554461s 2024-03-29T10:58:49.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.290+0000 7fa936ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa93806d9d0 0x7fa93813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46889/0 (socket says 172.21.15.107:46889) 2024-03-29T10:58:49.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.290+0000 7fa936ffd640 1 -- 172.21.15.107:0/1235451565 learned_addr learned my addr 172.21.15.107:0/1235451565 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:49.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 1 -- 172.21.15.107:0/1235451565 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310855235 0 0) 0x7fa9380aed10 con 0x7fa93806d9d0 2024-03-29T10:58:49.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:49.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:49.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:49.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:58:49.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 1 -- 172.21.15.107:0/1235451565 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa924003520 con 0x7fa93806d9d0 2024-03-29T10:58:49.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 1 -- 172.21.15.107:0/1235451565 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4014033413 0 0) 0x7fa924003520 con 0x7fa93806d9d0 2024-03-29T10:58:49.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 1 -- 172.21.15.107:0/1235451565 >> v1:172.21.15.107:6790/0 conn(0x7fa93813ef30 legacy=0x7fa9380af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:49.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 1 -- 172.21.15.107:0/1235451565 >> v1:172.21.15.129:6789/0 conn(0x7fa9380b07d0 legacy=0x7fa93814bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:49.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:49.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 1 monclient: found mon.noname-a 2024-03-29T10:58:49.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:49.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T10:58:49.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 1 -- 172.21.15.107:0/1235451565 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9380ab490 con 0x7fa93806d9d0 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 1 -- 172.21.15.107:0/1235451565 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa9280019f0 con 0x7fa93806d9d0 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.291+0000 7fa935ffb640 10 monclient: dump: 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:49.273 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:49.274 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:49.274 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:49.274 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:49.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa93d94d640 5 monclient: authenticate success, global_id 4290 2024-03-29T10:58:49.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa93d94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:49.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa935ffb640 1 -- 172.21.15.107:0/1235451565 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa928001c50 con 0x7fa93806d9d0 2024-03-29T10:58:49.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa935ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:49.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa935ffb640 1 -- 172.21.15.107:0/1235451565 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa9280019f0 con 0x7fa93806d9d0 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa935ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa935ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa935ffb640 10 monclient: dump: 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa93d94d640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:49.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.292+0000 7fa93d94d640 4 set_mon_vals no callback set 2024-03-29T10:58:49.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.293+0000 7fa93d94d640 10 monclient: shutdown 2024-03-29T10:58:49.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.293+0000 7fa93d94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:49.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.293+0000 7fa93d94d640 1 -- 172.21.15.107:0/1235451565 >> v1:172.21.15.107:6789/0 conn(0x7fa93806d9d0 legacy=0x7fa93813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.294+0000 7fa93d94d640 1 -- 172.21.15.107:0/1235451565 shutdown_connections 2024-03-29T10:58:49.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.294+0000 7fa93d94d640 1 -- 172.21.15.107:0/1235451565 >> 172.21.15.107:0/1235451565 conn(0x7fa93806b660 msgr2=0x7fa93806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:49.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.294+0000 7fa93d94d640 1 -- 172.21.15.107:0/1235451565 shutdown_connections 2024-03-29T10:58:49.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.294+0000 7fa93d94d640 1 -- 172.21.15.107:0/1235451565 wait complete. 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.295+0000 7fa93d94d640 10 monclient: build_initial_monmap 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.295+0000 7fa93d94d640 10 monclient: monmap: 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:49.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.295+0000 7fa93d94d640 1 librados: starting msgr at 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.295+0000 7fa93d94d640 1 librados: starting objecter 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.295+0000 7fa93d94d640 1 Processor -- start 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.296+0000 7fa93d94d640 1 -- start start 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.296+0000 7fa93d94d640 1 librados: setting wanted keys 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.296+0000 7fa93d94d640 1 librados: calling monclient init 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.296+0000 7fa93d94d640 10 monclient: init 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.296+0000 7fa93d94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.296+0000 7fa93d94d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.296+0000 7fa93d94d640 10 monclient(hunting): picked mon.noname-c con 0x7fa93806d9d0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.297+0000 7fa93d94d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9380b07d0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.297+0000 7fa93d94d640 10 monclient(hunting): picked mon.noname-b con 0x7fa93813ef30 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.297+0000 7fa93d94d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93806f130 con 0x7fa9380b07d0 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.297+0000 7fa93d94d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93806e590 con 0x7fa93813ef30 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.297+0000 7fa93d94d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9381dc800 con 0x7fa93806d9d0 2024-03-29T10:58:49.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.297+0000 7fa93d94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:49.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.297+0000 7fa93d94d640 10 monclient(hunting): authenticate will time out at 860.565031s 2024-03-29T10:58:49.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.301+0000 7fa9377fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa93813ef30 0x7fa93806f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:45295/0 (socket says 172.21.15.107:45295) 2024-03-29T10:58:49.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.301+0000 7fa9377fe640 1 -- 172.21.15.107:0/3208562206 learned_addr learned my addr 172.21.15.107:0/3208562206 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:49.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.301+0000 7fa92ffff640 10 client.?.objecter ms_handle_connect 0x7fa93813ef30 2024-03-29T10:58:49.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.301+0000 7fa92ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:49.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.302+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514541342 0 0) 0x7fa93806e590 con 0x7fa93813ef30 2024-03-29T10:58:49.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.302+0000 7fa92ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:49.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.302+0000 7fa92ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:49.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.302+0000 7fa92ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:49.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.302+0000 7fa92ffff640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:58:49.283 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.302+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa918004eb0 con 0x7fa93813ef30 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.302+0000 7fa92ffff640 10 client.?.objecter ms_handle_connect 0x7fa93806d9d0 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1824759287 0 0) 0x7fa918004eb0 con 0x7fa93813ef30 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa93806e590 con 0x7fa93813ef30 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa930004000 con 0x7fa93813ef30 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 10 monclient(hunting): dump: 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:49.285 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729808969 0 0) 0x7fa9381dc800 con 0x7fa93806d9d0 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:49.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:58:49.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.304+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9180056b0 con 0x7fa93806d9d0 2024-03-29T10:58:49.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.305+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2814113522 0 0) 0x7fa9180056b0 con 0x7fa93806d9d0 2024-03-29T10:58:49.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.305+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa918005b40 con 0x7fa93806d9d0 2024-03-29T10:58:49.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.305+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1576446632 0 0) 0x7fa93806e590 con 0x7fa93813ef30 2024-03-29T10:58:49.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.305+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 >> v1:172.21.15.107:6790/0 conn(0x7fa93806d9d0 legacy=0x7fa9381db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.305+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 >> v1:172.21.15.107:6789/0 conn(0x7fa9380b07d0 legacy=0x7fa93806e820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa92ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa92ffff640 1 monclient: found mon.b 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa92ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa92ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938071080 con 0x7fa93813ef30 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa92ffff640 10 monclient: _finish_auth 0 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa92ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa92ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa93d94d640 5 monclient: authenticate success, global_id 4279 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa93d94d640 10 monclient: _renew_subs 2024-03-29T10:58:49.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa93d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:49.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa938071330 con 0x7fa93813ef30 2024-03-29T10:58:49.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa93d94d640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:49.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa93d94d640 10 monclient: _renew_subs 2024-03-29T10:58:49.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa93d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:49.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa938152e80 con 0x7fa93813ef30 2024-03-29T10:58:49.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.307+0000 7fa93d94d640 1 librados: init done 2024-03-29T10:58:49.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa93d94d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:49.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa93d94d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa93d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa938138970 con 0x7fa93813ef30 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa930000ed0 con 0x7fa93813ef30 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa92ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa930004000 con 0x7fa93813ef30 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa92ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa92ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa92ffff640 10 monclient: dump: 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:49.290 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:49.291 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:49.291 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:49.291 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:49.291 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:49.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.309+0000 7fa934ff9640 4 set_mon_vals no callback set 2024-03-29T10:58:49.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.312+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fa930001180 con 0x7fa93813ef30 2024-03-29T10:58:49.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.314+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fa930084610 con 0x7fa93813ef30 2024-03-29T10:58:49.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.314+0000 7fa92ffff640 10 client.4279.objecter ms_dispatch 0x7fa938000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:49.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.314+0000 7fa92ffff640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:49.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.314+0000 7fa92ffff640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:49.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.314+0000 7fa92ffff640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T10:58:49.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.314+0000 7fa936ffd640 10 monclient: get_auth_request con 0x7fa91806e2e0 auth_method 0 2024-03-29T10:58:49.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.315+0000 7fa92ffff640 10 client.4279.objecter ms_handle_connect 0x7fa91806e2e0 2024-03-29T10:58:49.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.319+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa930050030 con 0x7fa93813ef30 2024-03-29T10:58:49.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.319+0000 7fa92ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:49.299 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.319+0000 7fa92ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:49.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.636+0000 7fa93d94d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:49.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.636+0000 7fa93d94d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:49.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.636+0000 7fa93d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:49.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.636+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa938070d40 con 0x7fa93813ef30 2024-03-29T10:58:49.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.638+0000 7fa92ffff640 1 -- 172.21.15.107:0/3208562206 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fa930051f60 con 0x7fa93813ef30 2024-03-29T10:58:49.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.638+0000 7fa92ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:49.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.638+0000 7fa92ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:49.618 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:58:49.618 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 10 librados: watch_flush enter 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 10 librados: watch_flush exit 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 10 client.4279.objecter successfully canceled tick 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 >> v1:172.21.15.107:6816/882194755 conn(0x7fa91806e2e0 legacy=0x7fa9180707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 10 monclient: shutdown 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 >> v1:172.21.15.129:6789/0 conn(0x7fa93813ef30 legacy=0x7fa93806f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 shutdown_connections 2024-03-29T10:58:49.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.641+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 >> 172.21.15.107:0/3208562206 conn(0x7fa93806b660 msgr2=0x7fa93813ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:49.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.644+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 shutdown_connections 2024-03-29T10:58:49.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.644+0000 7fa93d94d640 1 -- 172.21.15.107:0/3208562206 wait complete. 2024-03-29T10:58:49.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:49.644+0000 7fa93d94d640 1 librados: shutdown 2024-03-29T10:58:49.644 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:56.450 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:56.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.626+0000 7f37607fb640 10 monclient: get_monmap_and_config 2024-03-29T10:58:56.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.626+0000 7f37607fb640 10 monclient: build_initial_monmap 2024-03-29T10:58:56.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.626+0000 7f37607fb640 10 monclient: monmap: 2024-03-29T10:58:56.611 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:56.611 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:58:56.627642+0000 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:58:56.627642+0000 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.627+0000 7f37607fb640 1 Processor -- start 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.627+0000 7f37607fb640 1 -- start start 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.627+0000 7f37607fb640 10 monclient: init 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:56.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 10 monclient(hunting): picked mon.noname-a con 0x7f37580ad6c0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:56.613 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 10 monclient(hunting): picked mon.noname-c con 0x7f37580af6c0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:56.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 10 monclient(hunting): picked mon.noname-b con 0x7f375814fdc0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:56.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37580a94e0 con 0x7f37580ad6c0 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37580af3a0 con 0x7f37580af6c0 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37580afb60 con 0x7f375814fdc0 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.628+0000 7f37607fb640 10 monclient(hunting): authenticate will time out at 867.896648s 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.633+0000 7f375ed98640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f375814fdc0 0x7f375815a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:32951/0 (socket says 172.21.15.107:32951) 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.633+0000 7f375ed98640 1 -- 172.21.15.107:0/3556523719 learned_addr learned my addr 172.21.15.107:0/3556523719 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.633+0000 7f375d595640 1 -- 172.21.15.107:0/3556523719 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293644406 0 0) 0x7f37580afb60 con 0x7f375814fdc0 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.633+0000 7f375d595640 10 monclient(hunting): _negotiate 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.633+0000 7f375d595640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.633+0000 7f375d595640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.633+0000 7f375d595640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:58:56.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.633+0000 7f375d595640 1 -- 172.21.15.107:0/3556523719 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3744003520 con 0x7f375814fdc0 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 1 -- 172.21.15.107:0/3556523719 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2803689438 0 0) 0x7f3744003520 con 0x7f375814fdc0 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 1 -- 172.21.15.107:0/3556523719 >> v1:172.21.15.129:6789/0 conn(0x7f37580af6c0 legacy=0x7f3758152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 1 -- 172.21.15.107:0/3556523719 >> v1:172.21.15.107:6789/0 conn(0x7f37580ad6c0 legacy=0x7f375814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 10 monclient: _finish_hunting 0 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 1 monclient: found mon.noname-b 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 1 -- 172.21.15.107:0/3556523719 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580aab80 con 0x7f375814fdc0 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 10 monclient: _finish_auth 0 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 10 monclient: _check_auth_tickets 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 1 -- 172.21.15.107:0/3556523719 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f37540012f0 con 0x7f375814fdc0 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 10 monclient: dump: 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f37607fb640 5 monclient: authenticate success, global_id 4295 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f37607fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:56.619 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 1 -- 172.21.15.107:0/3556523719 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37540014c0 con 0x7f375814fdc0 2024-03-29T10:58:56.620 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f375d595640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:56.620 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f37607fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:56.620 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.634+0000 7f37607fb640 4 set_mon_vals no callback set 2024-03-29T10:58:56.620 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.635+0000 7f37607fb640 10 monclient: shutdown 2024-03-29T10:58:56.620 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.635+0000 7f37607fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:56.620 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.635+0000 7f37607fb640 1 -- 172.21.15.107:0/3556523719 >> v1:172.21.15.107:6790/0 conn(0x7f375814fdc0 legacy=0x7f375815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.621 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.637+0000 7f37607fb640 1 -- 172.21.15.107:0/3556523719 shutdown_connections 2024-03-29T10:58:56.621 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.637+0000 7f37607fb640 1 -- 172.21.15.107:0/3556523719 >> 172.21.15.107:0/3556523719 conn(0x7f37580b35e0 msgr2=0x7f37580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:56.621 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.637+0000 7f37607fb640 1 -- 172.21.15.107:0/3556523719 shutdown_connections 2024-03-29T10:58:56.621 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.637+0000 7f37607fb640 1 -- 172.21.15.107:0/3556523719 wait complete. 2024-03-29T10:58:56.621 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.638+0000 7f37607fb640 10 monclient: build_initial_monmap 2024-03-29T10:58:56.621 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.638+0000 7f37607fb640 10 monclient: monmap: 2024-03-29T10:58:56.621 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:58:56.621 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:56.621 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.638+0000 7f37607fb640 1 librados: starting msgr at 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.638+0000 7f37607fb640 1 librados: starting objecter 2024-03-29T10:58:56.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.638+0000 7f37607fb640 1 Processor -- start 2024-03-29T10:58:56.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.638+0000 7f37607fb640 1 -- start start 2024-03-29T10:58:56.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.638+0000 7f37607fb640 1 librados: setting wanted keys 2024-03-29T10:58:56.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.638+0000 7f37607fb640 1 librados: calling monclient init 2024-03-29T10:58:56.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.638+0000 7f37607fb640 10 monclient: init 2024-03-29T10:58:56.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.639+0000 7f37607fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:56.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.639+0000 7f37607fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:56.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.639+0000 7f37607fb640 10 monclient(hunting): picked mon.noname-b con 0x7f37580ad6c0 addr v1:172.21.15.129:6789/0 2024-03-29T10:58:56.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.639+0000 7f37607fb640 10 monclient(hunting): picked mon.noname-c con 0x7f37580af6c0 addr v1:172.21.15.107:6790/0 2024-03-29T10:58:56.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.644+0000 7f37607fb640 10 monclient(hunting): picked mon.noname-a con 0x7f375814fdc0 addr v1:172.21.15.107:6789/0 2024-03-29T10:58:56.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.644+0000 7f37607fb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37581f01b0 con 0x7f375814fdc0 2024-03-29T10:58:56.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.644+0000 7f37607fb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3758157a80 con 0x7f37580ad6c0 2024-03-29T10:58:56.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.644+0000 7f37607fb640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37581efcd0 con 0x7f37580af6c0 2024-03-29T10:58:56.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.644+0000 7f37607fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:56.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.644+0000 7f37607fb640 10 monclient(hunting): authenticate will time out at 867.912168s 2024-03-29T10:58:56.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f375e597640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f37580ad6c0 0x7f37581f0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:50535/0 (socket says 172.21.15.107:50535) 2024-03-29T10:58:56.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f375e597640 1 -- 172.21.15.107:0/167637247 learned_addr learned my addr 172.21.15.107:0/167637247 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f374effd640 10 client.?.objecter ms_handle_connect 0x7f37580ad6c0 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f374effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877551808 0 0) 0x7f3758157a80 con 0x7f37580ad6c0 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f374effd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f374effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f374effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f374effd640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.645+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f373c004eb0 con 0x7f37580ad6c0 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.646+0000 7f374effd640 10 client.?.objecter ms_handle_connect 0x7f375814fdc0 2024-03-29T10:58:56.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.646+0000 7f374effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.646+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3738470343 0 0) 0x7f373c004eb0 con 0x7f37580ad6c0 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3758157a80 con 0x7f37580ad6c0 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700156524 0 0) 0x7f37581f01b0 con 0x7f375814fdc0 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f373c004eb0 con 0x7f375814fdc0 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3748001960 con 0x7f37580ad6c0 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): dump: 2024-03-29T10:58:56.634 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3141891307 0 0) 0x7f373c004eb0 con 0x7f375814fdc0 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37581f01b0 con 0x7f375814fdc0 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f37540011b0 con 0x7f375814fdc0 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.635 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient(hunting): dump: 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1701716810 0 0) 0x7f3758157a80 con 0x7f37580ad6c0 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 >> v1:172.21.15.107:6790/0 conn(0x7f37580af6c0 legacy=0x7f37581ec7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.636 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 >> v1:172.21.15.107:6789/0 conn(0x7f375814fdc0 legacy=0x7f37581f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 monclient: found mon.b 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37581efe60 con 0x7f37580ad6c0 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient: _finish_auth 0 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f37607fb640 5 monclient: authenticate success, global_id 4282 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f37607fb640 10 monclient: _renew_subs 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f37607fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f37607fb640 1 -- 172.21.15.107:0/167637247 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375815b5c0 con 0x7f37580ad6c0 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f37607fb640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:56.637 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f37607fb640 10 monclient: _renew_subs 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f37607fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f37607fb640 1 -- 172.21.15.107:0/167637247 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f375815b910 con 0x7f37580ad6c0 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f37607fb640 1 librados: init done 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3748001b80 con 0x7f37580ad6c0 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3748003a40 con 0x7f37580ad6c0 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f374effd640 10 monclient: dump: 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:58:56.638 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.647+0000 7f375cd94640 4 set_mon_vals no callback set 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.649+0000 7f374cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.649+0000 7f374cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.649+0000 7f374cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.649+0000 7f374cff9640 1 -- 172.21.15.107:0/167637247 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3758072310 con 0x7f37580ad6c0 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.650+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f3748003ba0 con 0x7f37580ad6c0 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.650+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f3748084b00 con 0x7f37580ad6c0 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.650+0000 7f374effd640 10 client.4282.objecter ms_dispatch 0x7f3758000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:56.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.650+0000 7f374effd640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.650+0000 7f374effd640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.650+0000 7f374effd640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T10:58:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.652+0000 7f375dd96640 10 monclient: get_auth_request con 0x7f373c06de90 auth_method 0 2024-03-29T10:58:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.654+0000 7f374effd640 10 client.4282.objecter ms_handle_connect 0x7f373c06de90 2024-03-29T10:58:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.658+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37480503e0 con 0x7f37580ad6c0 2024-03-29T10:58:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.658+0000 7f374effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:56.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.658+0000 7f374effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:56.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.979+0000 7f374cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.979+0000 7f374cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.979+0000 7f374cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:58:56.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.979+0000 7f374cff9640 1 -- 172.21.15.107:0/167637247 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37581578a0 con 0x7f37580ad6c0 2024-03-29T10:58:56.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.986+0000 7f374effd640 1 -- 172.21.15.107:0/167637247 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f3748052310 con 0x7f37580ad6c0 2024-03-29T10:58:56.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.986+0000 7f374effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.986+0000 7f374effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:56.966 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:58:56.966 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.989+0000 7f374cff9640 10 librados: watch_flush enter 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.990+0000 7f374cff9640 10 librados: watch_flush exit 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.990+0000 7f374cff9640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.990+0000 7f374cff9640 10 client.4282.objecter successfully canceled tick 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.990+0000 7f374cff9640 1 -- 172.21.15.107:0/167637247 >> v1:172.21.15.107:6816/882194755 conn(0x7f373c06de90 legacy=0x7f373c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.990+0000 7f374cff9640 10 monclient: shutdown 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.990+0000 7f374cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.990+0000 7f374cff9640 1 -- 172.21.15.107:0/167637247 >> v1:172.21.15.129:6789/0 conn(0x7f37580ad6c0 legacy=0x7f37581f0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.990+0000 7f374cff9640 1 -- 172.21.15.107:0/167637247 shutdown_connections 2024-03-29T10:58:56.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.990+0000 7f374cff9640 1 -- 172.21.15.107:0/167637247 >> 172.21.15.107:0/167637247 conn(0x7f37580b35e0 msgr2=0x7f37580b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:56.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.991+0000 7f374cff9640 1 -- 172.21.15.107:0/167637247 shutdown_connections 2024-03-29T10:58:56.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.991+0000 7f374cff9640 1 -- 172.21.15.107:0/167637247 wait complete. 2024-03-29T10:58:56.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:58:56.991+0000 7f374cff9640 1 librados: shutdown 2024-03-29T10:58:56.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:03.797 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:03.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.971+0000 7f8f29e74640 10 monclient: get_monmap_and_config 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.971+0000 7f8f29e74640 10 monclient: build_initial_monmap 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.971+0000 7f8f29e74640 10 monclient: monmap: 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:59:03.972580+0000 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:59:03.972580+0000 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:59:03.952 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:59:03.953 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:03.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.972+0000 7f8f29e74640 1 Processor -- start 2024-03-29T10:59:03.953 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.972+0000 7f8f29e74640 1 -- start start 2024-03-29T10:59:03.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.972+0000 7f8f29e74640 10 monclient: init 2024-03-29T10:59:03.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.973+0000 7f8f29e74640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:03.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.973+0000 7f8f29e74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:03.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.973+0000 7f8f29e74640 10 monclient(hunting): picked mon.noname-c con 0x7f8f1c006a70 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:03.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.973+0000 7f8f29e74640 10 monclient(hunting): picked mon.noname-b con 0x7f8f1c02b100 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:03.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.978+0000 7f8f29e74640 10 monclient(hunting): picked mon.noname-a con 0x7f8f1c029c30 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.978+0000 7f8f29e74640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c02a270 con 0x7f8f1c029c30 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.978+0000 7f8f29e74640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c02c440 con 0x7f8f1c006a70 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.978+0000 7f8f29e74640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0c9440 con 0x7f8f1c02b100 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.978+0000 7f8f29e74640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:03.959 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.978+0000 7f8f29e74640 10 monclient(hunting): authenticate will time out at 875.246803s 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.979+0000 7f8f23fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f8f1c02b100 0x7f8f1c0cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34673/0 (socket says 172.21.15.107:34673) 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.979+0000 7f8f23fff640 1 -- 172.21.15.107:0/3065649031 learned_addr learned my addr 172.21.15.107:0/3065649031 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.979+0000 7f8f237fe640 1 -- 172.21.15.107:0/3065649031 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415132622 0 0) 0x7f8f1c0c9440 con 0x7f8f1c02b100 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.979+0000 7f8f237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.979+0000 7f8f237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.979+0000 7f8f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.979+0000 7f8f237fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.979+0000 7f8f237fe640 1 -- 172.21.15.107:0/3065649031 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f14003520 con 0x7f8f1c02b100 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.980+0000 7f8f237fe640 1 -- 172.21.15.107:0/3065649031 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 614819459 0 0) 0x7f8f14003520 con 0x7f8f1c02b100 2024-03-29T10:59:03.960 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.980+0000 7f8f237fe640 1 -- 172.21.15.107:0/3065649031 >> v1:172.21.15.129:6789/0 conn(0x7f8f1c006a70 legacy=0x7f8f1c0c8d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.980+0000 7f8f237fe640 1 -- 172.21.15.107:0/3065649031 >> v1:172.21.15.107:6789/0 conn(0x7f8f1c029c30 legacy=0x7f8f1c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 1 monclient: found mon.noname-b 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 1 -- 172.21.15.107:0/3065649031 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c0c95d0 con 0x7f8f1c02b100 2024-03-29T10:59:03.963 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 1 -- 172.21.15.107:0/3065649031 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f8f10001690 con 0x7f8f1c02b100 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.981+0000 7f8f237fe640 10 monclient: dump: 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:03.964 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.982+0000 7f8f29e74640 5 monclient: authenticate success, global_id 4298 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.982+0000 7f8f29e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.982+0000 7f8f237fe640 1 -- 172.21.15.107:0/3065649031 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f10001860 con 0x7f8f1c02b100 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.982+0000 7f8f237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.982+0000 7f8f237fe640 1 -- 172.21.15.107:0/3065649031 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f8f10001cb0 con 0x7f8f1c02b100 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.982+0000 7f8f29e74640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.982+0000 7f8f29e74640 4 set_mon_vals no callback set 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.983+0000 7f8f29e74640 10 monclient: shutdown 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.983+0000 7f8f29e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.983+0000 7f8f29e74640 1 -- 172.21.15.107:0/3065649031 >> v1:172.21.15.107:6790/0 conn(0x7f8f1c02b100 legacy=0x7f8f1c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.965 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.983+0000 7f8f237fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:59:03.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.984+0000 7f8f29e74640 1 -- 172.21.15.107:0/3065649031 shutdown_connections 2024-03-29T10:59:03.966 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.984+0000 7f8f29e74640 1 -- 172.21.15.107:0/3065649031 >> 172.21.15.107:0/3065649031 conn(0x7f8f1c0058f0 msgr2=0x7f8f1c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:03.967 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.987+0000 7f8f29e74640 1 -- 172.21.15.107:0/3065649031 shutdown_connections 2024-03-29T10:59:03.968 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.987+0000 7f8f29e74640 1 -- 172.21.15.107:0/3065649031 wait complete. 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.988+0000 7f8f29e74640 10 monclient: build_initial_monmap 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.989+0000 7f8f29e74640 10 monclient: monmap: 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:03.970 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:59:03.971 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:59:03.971 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:03.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.989+0000 7f8f29e74640 1 librados: starting msgr at 2024-03-29T10:59:03.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.989+0000 7f8f29e74640 1 librados: starting objecter 2024-03-29T10:59:03.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.989+0000 7f8f29e74640 1 Processor -- start 2024-03-29T10:59:03.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.990+0000 7f8f29e74640 1 -- start start 2024-03-29T10:59:03.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.990+0000 7f8f29e74640 1 librados: setting wanted keys 2024-03-29T10:59:03.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.990+0000 7f8f29e74640 1 librados: calling monclient init 2024-03-29T10:59:03.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.990+0000 7f8f29e74640 10 monclient: init 2024-03-29T10:59:03.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.991+0000 7f8f29e74640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:03.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.991+0000 7f8f29e74640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:03.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.991+0000 7f8f29e74640 10 monclient(hunting): picked mon.noname-b con 0x7f8f1c006a70 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:03.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.995+0000 7f8f29e74640 10 monclient(hunting): picked mon.noname-a con 0x7f8f1c029c30 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:03.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.997+0000 7f8f29e74640 10 monclient(hunting): picked mon.noname-c con 0x7f8f1c161130 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:03.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.997+0000 7f8f29e74640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c15e7c0 con 0x7f8f1c029c30 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.997+0000 7f8f29e74640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c02f5d0 con 0x7f8f1c006a70 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.997+0000 7f8f29e74640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c160c10 con 0x7f8f1c161130 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.997+0000 7f8f29e74640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.997+0000 7f8f29e74640 10 monclient(hunting): authenticate will time out at 875.265575s 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.998+0000 7f8f23fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8f1c029c30 0x7f8f1c02d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41601/0 (socket says 172.21.15.107:41601) 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.998+0000 7f8f23fff640 1 -- 172.21.15.107:0/3304613215 learned_addr learned my addr 172.21.15.107:0/3304613215 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.997+0000 7f8f28e72640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f8f1c006a70 0x7f8f1c02def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:57241/0 (socket says 172.21.15.107:57241) 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.998+0000 7f8f217fa640 10 client.?.objecter ms_handle_connect 0x7f8f1c006a70 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.998+0000 7f8f217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.998+0000 7f8f217fa640 10 client.?.objecter ms_handle_connect 0x7f8f1c029c30 2024-03-29T10:59:03.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:03.998+0000 7f8f217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874603939 0 0) 0x7f8f1c02f5d0 con 0x7f8f1c006a70 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f08004eb0 con 0x7f8f1c006a70 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122161399 0 0) 0x7f8f1c15e7c0 con 0x7f8f1c029c30 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.983 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c02f5d0 con 0x7f8f1c029c30 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 client.?.objecter ms_handle_connect 0x7f8f1c161130 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 755788366 0 0) 0x7f8f1c02f5d0 con 0x7f8f1c029c30 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1c15e7c0 con 0x7f8f1c029c30 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f8f10001b70 con 0x7f8f1c029c30 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): dump: 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:03.984 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.001+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264127938 0 0) 0x7f8f1c160c10 con 0x7f8f1c161130 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:59:03.986 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c02f5d0 con 0x7f8f1c161130 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2419507454 0 0) 0x7f8f1c15e7c0 con 0x7f8f1c029c30 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 >> v1:172.21.15.107:6790/0 conn(0x7f8f1c161130 legacy=0x7f8f1c031cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 >> v1:172.21.15.129:6789/0 conn(0x7f8f1c006a70 legacy=0x7f8f1c02def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 10 monclient: _finish_hunting 0 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 1 monclient: found mon.a 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c0cce60 con 0x7f8f1c029c30 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.002+0000 7f8f217fa640 10 monclient: _finish_auth 0 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f217fa640 10 monclient: _check_auth_tickets 2024-03-29T10:59:03.987 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f10004a90 con 0x7f8f1c029c30 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f8f10004da0 con 0x7f8f1c029c30 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f217fa640 10 monclient: dump: 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:03.988 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f29e74640 5 monclient: authenticate success, global_id 4296 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f29e74640 10 monclient: _renew_subs 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f29e74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f29e74640 1 -- 172.21.15.107:0/3304613215 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c0323e0 con 0x7f8f1c029c30 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f29e74640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f29e74640 10 monclient: _renew_subs 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f29e74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:03.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f29e74640 1 -- 172.21.15.107:0/3304613215 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c1615c0 con 0x7f8f1c029c30 2024-03-29T10:59:03.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.003+0000 7f8f29e74640 1 librados: init done 2024-03-29T10:59:03.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.004+0000 7f8f22ffd640 4 set_mon_vals no callback set 2024-03-29T10:59:03.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.005+0000 7f8f0effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.005+0000 7f8f0effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.005+0000 7f8f0effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:03.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.005+0000 7f8f0effd640 1 -- 172.21.15.107:0/3304613215 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c160da0 con 0x7f8f1c029c30 2024-03-29T10:59:03.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.009+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f8f1001ad70 con 0x7f8f1c029c30 2024-03-29T10:59:03.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.010+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f8f10084ac0 con 0x7f8f1c029c30 2024-03-29T10:59:03.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.010+0000 7f8f217fa640 10 client.4296.objecter ms_dispatch 0x7f8f24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:03.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.010+0000 7f8f217fa640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:03.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.010+0000 7f8f217fa640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:03.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.010+0000 7f8f28e72640 10 monclient: get_auth_request con 0x7f8f0806db10 auth_method 0 2024-03-29T10:59:03.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.010+0000 7f8f217fa640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T10:59:03.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.011+0000 7f8f217fa640 10 client.4296.objecter ms_handle_connect 0x7f8f0806db10 2024-03-29T10:59:04.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.019+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f100503a0 con 0x7f8f1c029c30 2024-03-29T10:59:04.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.019+0000 7f8f217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:04.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.019+0000 7f8f217fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:04.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.337+0000 7f8f0effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:04.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.337+0000 7f8f0effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:04.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.337+0000 7f8f0effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:04.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.337+0000 7f8f0effd640 1 -- 172.21.15.107:0/3304613215 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f1c0c49c0 con 0x7f8f1c029c30 2024-03-29T10:59:04.319 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:59:04.319 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:04.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.338+0000 7f8f217fa640 1 -- 172.21.15.107:0/3304613215 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f8f100522d0 con 0x7f8f1c029c30 2024-03-29T10:59:04.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.338+0000 7f8f217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:04.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.338+0000 7f8f217fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:04.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.342+0000 7f8f0effd640 10 librados: watch_flush enter 2024-03-29T10:59:04.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.342+0000 7f8f0effd640 10 librados: watch_flush exit 2024-03-29T10:59:04.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.342+0000 7f8f0effd640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T10:59:04.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.342+0000 7f8f0effd640 10 client.4296.objecter successfully canceled tick 2024-03-29T10:59:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.345+0000 7f8f0effd640 1 -- 172.21.15.107:0/3304613215 >> v1:172.21.15.107:6816/882194755 conn(0x7f8f0806db10 legacy=0x7f8f0806ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.345+0000 7f8f0effd640 10 monclient: shutdown 2024-03-29T10:59:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.345+0000 7f8f0effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:04.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.345+0000 7f8f0effd640 1 -- 172.21.15.107:0/3304613215 >> v1:172.21.15.107:6789/0 conn(0x7f8f1c029c30 legacy=0x7f8f1c02d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:04.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.346+0000 7f8f0effd640 1 -- 172.21.15.107:0/3304613215 shutdown_connections 2024-03-29T10:59:04.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.346+0000 7f8f0effd640 1 -- 172.21.15.107:0/3304613215 >> 172.21.15.107:0/3304613215 conn(0x7f8f1c0058f0 msgr2=0x7f8f1c0c3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:04.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.346+0000 7f8f0effd640 1 -- 172.21.15.107:0/3304613215 shutdown_connections 2024-03-29T10:59:04.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.346+0000 7f8f0effd640 1 -- 172.21.15.107:0/3304613215 wait complete. 2024-03-29T10:59:04.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:04.346+0000 7f8f0effd640 1 librados: shutdown 2024-03-29T10:59:04.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:11.154 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:11.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.326+0000 7fd59a4a8640 10 monclient: get_monmap_and_config 2024-03-29T10:59:11.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.326+0000 7fd59a4a8640 10 monclient: build_initial_monmap 2024-03-29T10:59:11.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.326+0000 7fd59a4a8640 10 monclient: monmap: 2024-03-29T10:59:11.307 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:59:11.327299+0000 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:59:11.327299+0000 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:11.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.328+0000 7fd59a4a8640 1 Processor -- start 2024-03-29T10:59:11.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.328+0000 7fd59a4a8640 1 -- start start 2024-03-29T10:59:11.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.328+0000 7fd59a4a8640 10 monclient: init 2024-03-29T10:59:11.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:11.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:59:11.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd5940a9ac0 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:11.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd59413ee90 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:11.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd5940b4100 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:11.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940af6f0 con 0x7fd5940a9ac0 2024-03-29T10:59:11.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940aac80 con 0x7fd59413ee90 2024-03-29T10:59:11.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940aaf60 con 0x7fd5940b4100 2024-03-29T10:59:11.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:11.310 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.329+0000 7fd59a4a8640 10 monclient(hunting): authenticate will time out at 882.597636s 2024-03-29T10:59:11.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.332+0000 7fd598a45640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd5940b4100 0x7fd59414a390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53709/0 (socket says 172.21.15.107:53709) 2024-03-29T10:59:11.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.332+0000 7fd598a45640 1 -- 172.21.15.107:0/1587388594 learned_addr learned my addr 172.21.15.107:0/1587388594 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:11.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.333+0000 7fd592ffd640 1 -- 172.21.15.107:0/1587388594 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934244654 0 0) 0x7fd5940aaf60 con 0x7fd5940b4100 2024-03-29T10:59:11.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.333+0000 7fd592ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:11.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.333+0000 7fd592ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.333+0000 7fd592ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.333+0000 7fd592ffd640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.333+0000 7fd592ffd640 1 -- 172.21.15.107:0/1587388594 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd57c003520 con 0x7fd5940b4100 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.334+0000 7fd592ffd640 1 -- 172.21.15.107:0/1587388594 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 715342293 0 0) 0x7fd57c003520 con 0x7fd5940b4100 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.334+0000 7fd592ffd640 1 -- 172.21.15.107:0/1587388594 >> v1:172.21.15.129:6789/0 conn(0x7fd59413ee90 legacy=0x7fd59413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.334+0000 7fd592ffd640 1 -- 172.21.15.107:0/1587388594 >> v1:172.21.15.107:6789/0 conn(0x7fd5940a9ac0 legacy=0x7fd5940adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 1 monclient: found mon.noname-b 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 1 -- 172.21.15.107:0/1587388594 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59413e970 con 0x7fd5940b4100 2024-03-29T10:59:11.317 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 1 -- 172.21.15.107:0/1587388594 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fd58c001a10 con 0x7fd5940b4100 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 10 monclient: dump: 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:11.318 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd59a4a8640 5 monclient: authenticate success, global_id 4304 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd59a4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 1 -- 172.21.15.107:0/1587388594 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd58c001ca0 con 0x7fd5940b4100 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd592ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd59a4a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.335+0000 7fd59a4a8640 4 set_mon_vals no callback set 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd59a4a8640 10 monclient: shutdown 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd592ffd640 1 -- 172.21.15.107:0/1587388594 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fd58c002030 con 0x7fd5940b4100 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd592ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:11.319 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd592ffd640 10 monclient: dump: 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd59a4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1587388594 >> v1:172.21.15.107:6790/0 conn(0x7fd5940b4100 legacy=0x7fd59414a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1587388594 shutdown_connections 2024-03-29T10:59:11.320 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1587388594 >> 172.21.15.107:0/1587388594 conn(0x7fd5940b35e0 msgr2=0x7fd5940b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:11.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1587388594 shutdown_connections 2024-03-29T10:59:11.321 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.336+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1587388594 wait complete. 2024-03-29T10:59:11.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.337+0000 7fd59a4a8640 10 monclient: build_initial_monmap 2024-03-29T10:59:11.323 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.337+0000 7fd59a4a8640 10 monclient: monmap: 2024-03-29T10:59:11.323 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:11.323 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:11.323 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:59:11.323 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:59:11.323 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:11.323 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:11.323 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.338+0000 7fd59a4a8640 1 librados: starting msgr at 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.338+0000 7fd59a4a8640 1 librados: starting objecter 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.338+0000 7fd59a4a8640 1 Processor -- start 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.338+0000 7fd59a4a8640 1 -- start start 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.338+0000 7fd59a4a8640 1 librados: setting wanted keys 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.338+0000 7fd59a4a8640 1 librados: calling monclient init 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.338+0000 7fd59a4a8640 10 monclient: init 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.339+0000 7fd59a4a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.339+0000 7fd59a4a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.339+0000 7fd59a4a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd5940a9ac0 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.339+0000 7fd59a4a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd5940b4100 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:11.324 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.342+0000 7fd59a4a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd59413ee90 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:11.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.342+0000 7fd59a4a8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd594140430 con 0x7fd59413ee90 2024-03-29T10:59:11.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.342+0000 7fd59a4a8640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd594142310 con 0x7fd5940b4100 2024-03-29T10:59:11.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.342+0000 7fd59a4a8640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59414f310 con 0x7fd5940a9ac0 2024-03-29T10:59:11.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.342+0000 7fd59a4a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:11.325 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.342+0000 7fd59a4a8640 10 monclient(hunting): authenticate will time out at 882.610637s 2024-03-29T10:59:11.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.344+0000 7fd598a45640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd59413ee90 0x7fd5941ef5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45609/0 (socket says 172.21.15.107:45609) 2024-03-29T10:59:11.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.344+0000 7fd598a45640 1 -- 172.21.15.107:0/1613350408 learned_addr learned my addr 172.21.15.107:0/1613350408 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:11.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.345+0000 7fd590ff9640 10 client.?.objecter ms_handle_connect 0x7fd59413ee90 2024-03-29T10:59:11.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.345+0000 7fd590ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:11.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.345+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489175707 0 0) 0x7fd594140430 con 0x7fd59413ee90 2024-03-29T10:59:11.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.345+0000 7fd590ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:11.326 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.345+0000 7fd590ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:11.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.345+0000 7fd590ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:11.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.345+0000 7fd590ff9640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:59:11.327 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.345+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd580004eb0 con 0x7fd59413ee90 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.349+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3782267116 0 0) 0x7fd580004eb0 con 0x7fd59413ee90 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.349+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd594140430 con 0x7fd59413ee90 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 10 client.?.objecter ms_handle_connect 0x7fd5940b4100 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fd58c0018d0 con 0x7fd59413ee90 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:11.333 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2516280029 0 0) 0x7fd594140430 con 0x7fd59413ee90 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 >> v1:172.21.15.107:6790/0 conn(0x7fd5940a9ac0 legacy=0x7fd594140a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 >> v1:172.21.15.129:6789/0 conn(0x7fd5940b4100 legacy=0x7fd594071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:59:11.334 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 1 monclient: found mon.a 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.350+0000 7fd590ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594141110 con 0x7fd59413ee90 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd590ff9640 10 monclient: _finish_auth 0 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd590ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd590ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd59a4a8640 5 monclient: authenticate success, global_id 4299 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd59a4a8640 10 monclient: _renew_subs 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd59a4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5941e3d50 con 0x7fd59413ee90 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd59a4a8640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd59a4a8640 10 monclient: _renew_subs 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd59a4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:11.335 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59412c880 con 0x7fd59413ee90 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd59a4a8640 1 librados: init done 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd58c001aa0 con 0x7fd59413ee90 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd590ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.351+0000 7fd5927fc640 4 set_mon_vals no callback set 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.352+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fd58c005150 con 0x7fd59413ee90 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.352+0000 7fd590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.352+0000 7fd590ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.352+0000 7fd590ff9640 10 monclient: dump: 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:11.336 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:11.337 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:11.337 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:11.337 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:11.337 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:11.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.355+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fd58c01b120 con 0x7fd59413ee90 2024-03-29T10:59:11.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.356+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fd58c084f20 con 0x7fd59413ee90 2024-03-29T10:59:11.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.356+0000 7fd590ff9640 10 client.4299.objecter ms_dispatch 0x7fd594000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:11.338 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.356+0000 7fd590ff9640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:11.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.356+0000 7fd593fff640 10 monclient: get_auth_request con 0x7fd58006dd40 auth_method 0 2024-03-29T10:59:11.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.356+0000 7fd590ff9640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:11.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.356+0000 7fd590ff9640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T10:59:11.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.357+0000 7fd590ff9640 10 client.4299.objecter ms_handle_connect 0x7fd58006dd40 2024-03-29T10:59:11.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.357+0000 7fd59a4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:11.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.357+0000 7fd59a4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:11.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.357+0000 7fd59a4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:11.339 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.357+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd574005080 con 0x7fd59413ee90 2024-03-29T10:59:11.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.366+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd58c050800 con 0x7fd59413ee90 2024-03-29T10:59:11.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.366+0000 7fd590ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:11.346 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.366+0000 7fd590ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:11.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.691+0000 7fd59a4a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:11.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.691+0000 7fd59a4a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:11.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.691+0000 7fd59a4a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T10:59:11.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.691+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd574005330 con 0x7fd59413ee90 2024-03-29T10:59:11.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.692+0000 7fd590ff9640 1 -- 172.21.15.107:0/1613350408 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fd58c052730 con 0x7fd59413ee90 2024-03-29T10:59:11.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.692+0000 7fd590ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:11.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.692+0000 7fd590ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:11.673 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:59:11.673 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:11.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.696+0000 7fd59a4a8640 10 librados: watch_flush enter 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.697+0000 7fd59a4a8640 10 librados: watch_flush exit 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.697+0000 7fd59a4a8640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.697+0000 7fd59a4a8640 10 client.4299.objecter successfully canceled tick 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.698+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 >> v1:172.21.15.107:6816/882194755 conn(0x7fd58006dd40 legacy=0x7fd580070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.698+0000 7fd59a4a8640 10 monclient: shutdown 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.698+0000 7fd59a4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.698+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 >> v1:172.21.15.107:6789/0 conn(0x7fd59413ee90 legacy=0x7fd5941ef5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.698+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 shutdown_connections 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.698+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 >> 172.21.15.107:0/1613350408 conn(0x7fd5940b35e0 msgr2=0x7fd5940ac7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.698+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 shutdown_connections 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.699+0000 7fd59a4a8640 1 -- 172.21.15.107:0/1613350408 wait complete. 2024-03-29T10:59:11.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:11.699+0000 7fd59a4a8640 1 librados: shutdown 2024-03-29T10:59:11.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:18.501 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:18.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.676+0000 7f52eb59e640 10 monclient: get_monmap_and_config 2024-03-29T10:59:18.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.676+0000 7f52eb59e640 10 monclient: build_initial_monmap 2024-03-29T10:59:18.657 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.676+0000 7f52eb59e640 10 monclient: monmap: 2024-03-29T10:59:18.657 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:18.657 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:18.658 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:59:18.678166+0000 2024-03-29T10:59:18.658 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:59:18.678166+0000 2024-03-29T10:59:18.658 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:18.658 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:18.658 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:18.658 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:59:18.658 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:59:18.658 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:18.658 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.677+0000 7f52eb59e640 1 Processor -- start 2024-03-29T10:59:18.659 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.677+0000 7f52eb59e640 1 -- start start 2024-03-29T10:59:18.659 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.677+0000 7f52eb59e640 10 monclient: init 2024-03-29T10:59:18.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:18.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:18.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f52ec0b2800 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:18.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f52ec0b3220 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:18.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f52ec1439c0 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:18.660 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0b3840 con 0x7f52ec0b2800 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec06f230 con 0x7f52ec1439c0 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec06ef50 con 0x7f52ec0b3220 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:18.661 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.678+0000 7f52eb59e640 10 monclient(hunting): authenticate will time out at 889.947532s 2024-03-29T10:59:18.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.685+0000 7f52e9d9b640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f52ec0b3220 0x7f52ec146710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41255/0 (socket says 172.21.15.107:41255) 2024-03-29T10:59:18.666 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.685+0000 7f52e9d9b640 1 -- 172.21.15.107:0/3008284975 learned_addr learned my addr 172.21.15.107:0/3008284975 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:18.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.686+0000 7f52e959a640 1 -- 172.21.15.107:0/3008284975 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901847545 0 0) 0x7f52ec06ef50 con 0x7f52ec0b3220 2024-03-29T10:59:18.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.686+0000 7f52e959a640 10 monclient(hunting): _negotiate 2024-03-29T10:59:18.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.686+0000 7f52e959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:18.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.686+0000 7f52e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:18.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.686+0000 7f52e959a640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:59:18.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.686+0000 7f52e959a640 1 -- 172.21.15.107:0/3008284975 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d4003520 con 0x7f52ec0b3220 2024-03-29T10:59:18.667 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.686+0000 7f52e959a640 1 -- 172.21.15.107:0/3008284975 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1900574269 0 0) 0x7f52d4003520 con 0x7f52ec0b3220 2024-03-29T10:59:18.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.686+0000 7f52e959a640 1 -- 172.21.15.107:0/3008284975 >> v1:172.21.15.129:6789/0 conn(0x7f52ec1439c0 legacy=0x7f52ec151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.668 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.686+0000 7f52e959a640 1 -- 172.21.15.107:0/3008284975 >> v1:172.21.15.107:6789/0 conn(0x7f52ec0b2800 legacy=0x7f52ec1432b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:18.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: _finish_hunting 0 2024-03-29T10:59:18.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 1 monclient: found mon.noname-b 2024-03-29T10:59:18.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:18.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:59:18.671 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 1 -- 172.21.15.107:0/3008284975 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec071d30 con 0x7f52ec0b3220 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: _finish_auth 0 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: _check_auth_tickets 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 1 -- 172.21.15.107:0/3008284975 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f52dc001690 con 0x7f52ec0b3220 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: dump: 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:18.672 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 1 -- 172.21.15.107:0/3008284975 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52dc001860 con 0x7f52ec0b3220 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 1 -- 172.21.15.107:0/3008284975 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f52dc001cb0 con 0x7f52ec0b3220 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.688+0000 7f52e959a640 10 monclient: dump: 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.673 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.689+0000 7f52eb59e640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.689+0000 7f52eb59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.689+0000 7f52eb59e640 4 set_mon_vals no callback set 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.689+0000 7f52eb59e640 10 monclient: shutdown 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.689+0000 7f52eb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:18.674 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.689+0000 7f52eb59e640 1 -- 172.21.15.107:0/3008284975 >> v1:172.21.15.107:6790/0 conn(0x7f52ec0b3220 legacy=0x7f52ec146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.675 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.694+0000 7f52eb59e640 1 -- 172.21.15.107:0/3008284975 shutdown_connections 2024-03-29T10:59:18.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.694+0000 7f52eb59e640 1 -- 172.21.15.107:0/3008284975 >> 172.21.15.107:0/3008284975 conn(0x7f52ec066a00 msgr2=0x7f52ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:18.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.695+0000 7f52eb59e640 1 -- 172.21.15.107:0/3008284975 shutdown_connections 2024-03-29T10:59:18.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.695+0000 7f52eb59e640 1 -- 172.21.15.107:0/3008284975 wait complete. 2024-03-29T10:59:18.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.696+0000 7f52eb59e640 10 monclient: build_initial_monmap 2024-03-29T10:59:18.680 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.696+0000 7f52eb59e640 10 monclient: monmap: 2024-03-29T10:59:18.680 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:18.680 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:18.680 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:59:18.680 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:59:18.680 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.696+0000 7f52eb59e640 1 librados: starting msgr at 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.696+0000 7f52eb59e640 1 librados: starting objecter 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 1 Processor -- start 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 1 -- start start 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 1 librados: setting wanted keys 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 1 librados: calling monclient init 2024-03-29T10:59:18.681 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 10 monclient: init 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f52ec0b2800 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f52ec0b3220 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f52ec1439c0 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec149f50 con 0x7f52ec1439c0 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0a9fd0 con 0x7f52ec0b3220 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec148de0 con 0x7f52ec0b2800 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:18.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.697+0000 7f52eb59e640 10 monclient(hunting): authenticate will time out at 889.966929s 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52ea59c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f52ec0b2800 0x7f52ec1e3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41373/0 (socket says 172.21.15.107:41373) 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52ea59c640 1 -- 172.21.15.107:0/3794455840 learned_addr learned my addr 172.21.15.107:0/3794455840 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52daffd640 10 client.?.objecter ms_handle_connect 0x7f52ec0b2800 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840782686 0 0) 0x7f52ec148de0 con 0x7f52ec0b2800 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52daffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52daffd640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:59:18.685 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.703+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52cc004eb0 con 0x7f52ec0b2800 2024-03-29T10:59:18.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.705+0000 7f52daffd640 10 client.?.objecter ms_handle_connect 0x7f52ec1439c0 2024-03-29T10:59:18.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.705+0000 7f52daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:18.686 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.705+0000 7f52daffd640 10 client.?.objecter ms_handle_connect 0x7f52ec0b3220 2024-03-29T10:59:18.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.705+0000 7f52daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:18.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.705+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1408470023 0 0) 0x7f52cc004eb0 con 0x7f52ec0b2800 2024-03-29T10:59:18.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52ec148de0 con 0x7f52ec0b2800 2024-03-29T10:59:18.687 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f52e0001960 con 0x7f52ec0b2800 2024-03-29T10:59:18.690 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.690 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:18.690 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 10 monclient(hunting): dump: 2024-03-29T10:59:18.690 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:18.690 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:18.690 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:18.690 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:18.690 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 516559410 0 0) 0x7f52ec148de0 con 0x7f52ec0b2800 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 >> v1:172.21.15.129:6789/0 conn(0x7f52ec0b3220 legacy=0x7f52ec1e42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 >> v1:172.21.15.107:6789/0 conn(0x7f52ec1439c0 legacy=0x7f52ec1ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 1 monclient: found mon.c 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.706+0000 7f52daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec148f70 con 0x7f52ec0b2800 2024-03-29T10:59:18.691 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52daffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52daffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52eb59e640 5 monclient: authenticate success, global_id 4310 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52eb59e640 10 monclient: _renew_subs 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52eb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec149910 con 0x7f52ec0b2800 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52eb59e640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52eb59e640 10 monclient: _renew_subs 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52eb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec0b6d90 con 0x7f52ec0b2800 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52eb59e640 1 librados: init done 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52e0001ac0 con 0x7f52ec0b2800 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.707+0000 7f52e8d99640 4 set_mon_vals no callback set 2024-03-29T10:59:18.692 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.708+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f52e0001d80 con 0x7f52ec0b2800 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.708+0000 7f52daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.708+0000 7f52daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.708+0000 7f52daffd640 10 monclient: dump: 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:18.693 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:18.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.710+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f52e0003a40 con 0x7f52ec0b2800 2024-03-29T10:59:18.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.711+0000 7f52e9d9b640 10 monclient: get_auth_request con 0x7f52cc06ddb0 auth_method 0 2024-03-29T10:59:18.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52daffd640 10 client.4310.objecter ms_handle_connect 0x7f52cc06ddb0 2024-03-29T10:59:18.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f52e0084a80 con 0x7f52ec0b2800 2024-03-29T10:59:18.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52daffd640 10 client.4310.objecter ms_dispatch 0x7f52ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:18.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52daffd640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:18.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52daffd640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:18.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52eb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:18.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52daffd640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T10:59:18.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52eb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:18.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52eb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:18.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.712+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d0005080 con 0x7f52ec0b2800 2024-03-29T10:59:18.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.724+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52e0050190 con 0x7f52ec0b2800 2024-03-29T10:59:18.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.725+0000 7f52daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:18.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:18.725+0000 7f52daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:19.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.052+0000 7f52eb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:19.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.052+0000 7f52eb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:19.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.052+0000 7f52eb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:19.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.052+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52d0005330 con 0x7f52ec0b2800 2024-03-29T10:59:19.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.053+0000 7f52daffd640 1 -- 172.21.15.107:0/3794455840 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f52e0052290 con 0x7f52ec0b2800 2024-03-29T10:59:19.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.053+0000 7f52daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:19.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.053+0000 7f52daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:19.035 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:59:19.035 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:19.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.057+0000 7f52eb59e640 10 librados: watch_flush enter 2024-03-29T10:59:19.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.057+0000 7f52eb59e640 10 librados: watch_flush exit 2024-03-29T10:59:19.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.057+0000 7f52eb59e640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T10:59:19.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.057+0000 7f52eb59e640 10 client.4310.objecter successfully canceled tick 2024-03-29T10:59:19.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.058+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 >> v1:172.21.15.107:6816/882194755 conn(0x7f52cc06ddb0 legacy=0x7f52cc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:19.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.058+0000 7f52eb59e640 10 monclient: shutdown 2024-03-29T10:59:19.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.058+0000 7f52eb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:19.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.058+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 >> v1:172.21.15.107:6790/0 conn(0x7f52ec0b2800 legacy=0x7f52ec1e3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:19.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.059+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 shutdown_connections 2024-03-29T10:59:19.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.059+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 >> 172.21.15.107:0/3794455840 conn(0x7f52ec066a00 msgr2=0x7f52ec1402b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:19.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.059+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 shutdown_connections 2024-03-29T10:59:19.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.059+0000 7f52eb59e640 1 -- 172.21.15.107:0/3794455840 wait complete. 2024-03-29T10:59:19.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:19.059+0000 7f52eb59e640 1 librados: shutdown 2024-03-29T10:59:19.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:25.868 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:26.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.040+0000 7fceaa606640 10 monclient: get_monmap_and_config 2024-03-29T10:59:26.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.040+0000 7fceaa606640 10 monclient: build_initial_monmap 2024-03-29T10:59:26.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.040+0000 7fceaa606640 10 monclient: monmap: 2024-03-29T10:59:26.022 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:26.022 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:26.022 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:59:26.041431+0000 2024-03-29T10:59:26.022 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:59:26.041431+0000 2024-03-29T10:59:26.023 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:26.023 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:26.023 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:26.023 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:59:26.023 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:59:26.023 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.043+0000 7fceaa606640 1 Processor -- start 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.043+0000 7fceaa606640 1 -- start start 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.043+0000 7fceaa606640 10 monclient: init 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 10 monclient(hunting): picked mon.noname-c con 0x7fcea406d9d0 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 10 monclient(hunting): picked mon.noname-a con 0x7fcea413ef30 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 10 monclient(hunting): picked mon.noname-b con 0x7fcea40b07d0 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40aed10 con 0x7fcea413ef30 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40b0c40 con 0x7fcea406d9d0 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40ab1b0 con 0x7fcea40b07d0 2024-03-29T10:59:26.024 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:26.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.044+0000 7fceaa606640 10 monclient(hunting): authenticate will time out at 897.312107s 2024-03-29T10:59:26.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.045+0000 7fcea8ba3640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcea40b07d0 0x7fcea414ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36628/0 (socket says 172.21.15.107:36628) 2024-03-29T10:59:26.025 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.045+0000 7fcea8ba3640 1 -- 172.21.15.107:0/470532046 learned_addr learned my addr 172.21.15.107:0/470532046 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:26.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.046+0000 7fcea2ffd640 1 -- 172.21.15.107:0/470532046 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934149269 0 0) 0x7fcea40ab1b0 con 0x7fcea40b07d0 2024-03-29T10:59:26.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.046+0000 7fcea2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:26.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.046+0000 7fcea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:26.026 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.046+0000 7fcea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:26.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.046+0000 7fcea2ffd640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:59:26.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.046+0000 7fcea2ffd640 1 -- 172.21.15.107:0/470532046 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c003520 con 0x7fcea40b07d0 2024-03-29T10:59:26.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.047+0000 7fcea2ffd640 1 -- 172.21.15.107:0/470532046 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2280169145 0 0) 0x7fce8c003520 con 0x7fcea40b07d0 2024-03-29T10:59:26.027 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.047+0000 7fcea2ffd640 1 -- 172.21.15.107:0/470532046 >> v1:172.21.15.129:6789/0 conn(0x7fcea406d9d0 legacy=0x7fcea413e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:26.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 1 -- 172.21.15.107:0/470532046 >> v1:172.21.15.107:6789/0 conn(0x7fcea413ef30 legacy=0x7fcea40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 1 monclient: found mon.noname-b 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 1 -- 172.21.15.107:0/470532046 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40ab490 con 0x7fcea40b07d0 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 1 -- 172.21.15.107:0/470532046 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fce94001a10 con 0x7fcea40b07d0 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: dump: 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:26.030 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fceaa606640 5 monclient: authenticate success, global_id 4313 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fceaa606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 1 -- 172.21.15.107:0/470532046 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce94001be0 con 0x7fcea40b07d0 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 1 -- 172.21.15.107:0/470532046 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fce94002030 con 0x7fcea40b07d0 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:26.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fcea2ffd640 10 monclient: dump: 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fceaa606640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:26.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.048+0000 7fceaa606640 4 set_mon_vals no callback set 2024-03-29T10:59:26.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.049+0000 7fceaa606640 10 monclient: shutdown 2024-03-29T10:59:26.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.049+0000 7fceaa606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:26.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.049+0000 7fceaa606640 1 -- 172.21.15.107:0/470532046 >> v1:172.21.15.107:6790/0 conn(0x7fcea40b07d0 legacy=0x7fcea414ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.049+0000 7fceaa606640 1 -- 172.21.15.107:0/470532046 shutdown_connections 2024-03-29T10:59:26.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.049+0000 7fceaa606640 1 -- 172.21.15.107:0/470532046 >> 172.21.15.107:0/470532046 conn(0x7fcea406b660 msgr2=0x7fcea406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:26.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.052+0000 7fceaa606640 1 -- 172.21.15.107:0/470532046 shutdown_connections 2024-03-29T10:59:26.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.053+0000 7fceaa606640 1 -- 172.21.15.107:0/470532046 wait complete. 2024-03-29T10:59:26.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.054+0000 7fceaa606640 10 monclient: build_initial_monmap 2024-03-29T10:59:26.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.054+0000 7fceaa606640 10 monclient: monmap: 2024-03-29T10:59:26.034 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:26.034 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:26.034 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:59:26.034 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:59:26.035 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:26.035 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:26.035 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:26.035 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:59:26.035 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:59:26.035 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:26.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.054+0000 7fceaa606640 1 librados: starting msgr at 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.054+0000 7fceaa606640 1 librados: starting objecter 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.055+0000 7fceaa606640 1 Processor -- start 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.055+0000 7fceaa606640 1 -- start start 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.055+0000 7fceaa606640 1 librados: setting wanted keys 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.055+0000 7fceaa606640 1 librados: calling monclient init 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.055+0000 7fceaa606640 10 monclient: init 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 10 monclient(hunting): picked mon.noname-a con 0x7fcea406d9d0 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 10 monclient(hunting): picked mon.noname-b con 0x7fcea40b07d0 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 10 monclient(hunting): picked mon.noname-c con 0x7fcea413ef30 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea41dbbf0 con 0x7fcea406d9d0 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea4137ea0 con 0x7fcea40b07d0 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea4135290 con 0x7fcea413ef30 2024-03-29T10:59:26.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:26.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.056+0000 7fceaa606640 10 monclient(hunting): authenticate will time out at 897.324922s 2024-03-29T10:59:26.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.057+0000 7fcea8ba3640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcea413ef30 0x7fcea4134b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37074/0 (socket says 172.21.15.107:37074) 2024-03-29T10:59:26.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.057+0000 7fcea8ba3640 1 -- 172.21.15.107:0/902953086 learned_addr learned my addr 172.21.15.107:0/902953086 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:26.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.057+0000 7fcea0ff9640 10 client.?.objecter ms_handle_connect 0x7fcea413ef30 2024-03-29T10:59:26.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.057+0000 7fcea0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:26.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.058+0000 7fcea0ff9640 10 client.?.objecter ms_handle_connect 0x7fcea406d9d0 2024-03-29T10:59:26.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.058+0000 7fcea0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:26.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330427409 0 0) 0x7fcea4135290 con 0x7fcea413ef30 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce90004eb0 con 0x7fcea413ef30 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528148286 0 0) 0x7fcea41dbbf0 con 0x7fcea406d9d0 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.059+0000 7fcea0ff9640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:59:26.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.060+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea4135290 con 0x7fcea406d9d0 2024-03-29T10:59:26.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.060+0000 7fcea0ff9640 10 client.?.objecter ms_handle_connect 0x7fcea40b07d0 2024-03-29T10:59:26.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.060+0000 7fcea0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.060+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 567108515 0 0) 0x7fce90004eb0 con 0x7fcea413ef30 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcea41dbbf0 con 0x7fcea413ef30 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fce94001ef0 con 0x7fcea413ef30 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:26.042 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 472996270 0 0) 0x7fcea4135290 con 0x7fcea406d9d0 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce90004eb0 con 0x7fcea406d9d0 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fce9c001790 con 0x7fcea406d9d0 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.043 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3017500568 0 0) 0x7fcea41dbbf0 con 0x7fcea413ef30 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 >> v1:172.21.15.129:6789/0 conn(0x7fcea40b07d0 legacy=0x7fcea4134470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 >> v1:172.21.15.107:6789/0 conn(0x7fcea406d9d0 legacy=0x7fcea41dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 monclient: found mon.c 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:26.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea41df340 con 0x7fcea413ef30 2024-03-29T10:59:26.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:59:26.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:59:26.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.061+0000 7fcea0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:26.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.062+0000 7fceaa606640 5 monclient: authenticate success, global_id 4316 2024-03-29T10:59:26.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.062+0000 7fceaa606640 10 monclient: _renew_subs 2024-03-29T10:59:26.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.062+0000 7fceaa606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:26.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.062+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea4135b20 con 0x7fcea413ef30 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.062+0000 7fceaa606640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.062+0000 7fceaa606640 10 monclient: _renew_subs 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.062+0000 7fceaa606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.062+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea4138780 con 0x7fcea413ef30 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.062+0000 7fceaa606640 1 librados: init done 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.065+0000 7fceaa606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.065+0000 7fceaa606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.065+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce94004f70 con 0x7fcea413ef30 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.065+0000 7fceaa606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.065+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea4075f10 con 0x7fcea413ef30 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.065+0000 7fcea0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.065+0000 7fcea1ffb640 4 set_mon_vals no callback set 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.066+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fce94005280 con 0x7fcea413ef30 2024-03-29T10:59:26.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.066+0000 7fcea0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.066+0000 7fcea0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.066+0000 7fcea0ff9640 10 monclient: dump: 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:26.048 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:26.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.067+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fce9401b250 con 0x7fcea413ef30 2024-03-29T10:59:26.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.068+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fce94084f40 con 0x7fcea413ef30 2024-03-29T10:59:26.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.068+0000 7fcea0ff9640 10 client.4316.objecter ms_dispatch 0x7fcea4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:26.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.068+0000 7fcea3fff640 10 monclient: get_auth_request con 0x7fce9006de10 auth_method 0 2024-03-29T10:59:26.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.068+0000 7fcea0ff9640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:26.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.068+0000 7fcea0ff9640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:26.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.068+0000 7fcea0ff9640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T10:59:26.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.068+0000 7fcea0ff9640 10 client.4316.objecter ms_handle_connect 0x7fce9006de10 2024-03-29T10:59:26.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.075+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce94050820 con 0x7fcea413ef30 2024-03-29T10:59:26.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.075+0000 7fcea0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:26.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.075+0000 7fcea0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:26.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.422+0000 7fceaa606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:26.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.422+0000 7fceaa606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:26.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.422+0000 7fceaa606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:26.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.422+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcea4135cb0 con 0x7fcea413ef30 2024-03-29T10:59:26.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.425+0000 7fcea0ff9640 1 -- 172.21.15.107:0/902953086 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fce94052750 con 0x7fcea413ef30 2024-03-29T10:59:26.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.425+0000 7fcea0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:26.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.425+0000 7fcea0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:26.410 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:59:26.410 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:26.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.433+0000 7fceaa606640 10 librados: watch_flush enter 2024-03-29T10:59:26.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.434+0000 7fceaa606640 10 librados: watch_flush exit 2024-03-29T10:59:26.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.434+0000 7fceaa606640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T10:59:26.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.434+0000 7fceaa606640 10 client.4316.objecter successfully canceled tick 2024-03-29T10:59:26.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.435+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 >> v1:172.21.15.107:6816/882194755 conn(0x7fce9006de10 legacy=0x7fce900702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.435+0000 7fceaa606640 10 monclient: shutdown 2024-03-29T10:59:26.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.435+0000 7fceaa606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:26.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.435+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 >> v1:172.21.15.107:6790/0 conn(0x7fcea413ef30 legacy=0x7fcea4134b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:26.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.437+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 shutdown_connections 2024-03-29T10:59:26.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.437+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 >> 172.21.15.107:0/902953086 conn(0x7fcea406b660 msgr2=0x7fcea406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:26.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.438+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 shutdown_connections 2024-03-29T10:59:26.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.438+0000 7fceaa606640 1 -- 172.21.15.107:0/902953086 wait complete. 2024-03-29T10:59:26.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:26.438+0000 7fceaa606640 1 librados: shutdown 2024-03-29T10:59:26.436 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:33.242 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.404+0000 7f568a8ed640 10 monclient: get_monmap_and_config 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.404+0000 7f568a8ed640 10 monclient: build_initial_monmap 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.404+0000 7f568a8ed640 10 monclient: monmap: 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:59:33.405875+0000 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:59:33.405875+0000 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:33.385 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:33.386 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:59:33.386 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:59:33.386 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:33.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.405+0000 7f568a8ed640 1 Processor -- start 2024-03-29T10:59:33.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.405+0000 7f568a8ed640 1 -- start start 2024-03-29T10:59:33.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.405+0000 7f568a8ed640 10 monclient: init 2024-03-29T10:59:33.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:33.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:33.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 10 monclient(hunting): picked mon.noname-c con 0x7f56840b0860 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:33.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 10 monclient(hunting): picked mon.noname-b con 0x7f5684072450 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:33.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 10 monclient(hunting): picked mon.noname-a con 0x7f568413ee40 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:33.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56840ad840 con 0x7f568413ee40 2024-03-29T10:59:33.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568413c830 con 0x7f56840b0860 2024-03-29T10:59:33.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568413c9c0 con 0x7f5684072450 2024-03-29T10:59:33.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:33.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.406+0000 7f568a8ed640 10 monclient(hunting): authenticate will time out at 904.674815s 2024-03-29T10:59:33.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.407+0000 7f5683fff640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f56840b0860 0x7f56840acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:56666/0 (socket says 172.21.15.107:56666) 2024-03-29T10:59:33.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.407+0000 7f5683fff640 1 -- 172.21.15.107:0/2342812899 learned_addr learned my addr 172.21.15.107:0/2342812899 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:33.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.407+0000 7f5682ffd640 1 -- 172.21.15.107:0/2342812899 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952349691 0 0) 0x7f56840ad840 con 0x7f568413ee40 2024-03-29T10:59:33.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.407+0000 7f5682ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.407+0000 7f5682ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.407+0000 7f5682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.407+0000 7f5682ffd640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.407+0000 7f5682ffd640 1 -- 172.21.15.107:0/2342812899 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566c003520 con 0x7f568413ee40 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 1 -- 172.21.15.107:0/2342812899 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3559095963 0 0) 0x7f566c003520 con 0x7f568413ee40 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 1 -- 172.21.15.107:0/2342812899 >> v1:172.21.15.107:6790/0 conn(0x7f5684072450 legacy=0x7f568413c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 1 -- 172.21.15.107:0/2342812899 >> v1:172.21.15.129:6789/0 conn(0x7f56840b0860 legacy=0x7f56840acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 1 monclient: found mon.noname-a 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T10:59:33.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 1 -- 172.21.15.107:0/2342812899 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568413f970 con 0x7f568413ee40 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 1 -- 172.21.15.107:0/2342812899 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f567c001a10 con 0x7f568413ee40 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f5682ffd640 10 monclient: dump: 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:33.392 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f568a8ed640 5 monclient: authenticate success, global_id 4308 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.408+0000 7f568a8ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f5682ffd640 1 -- 172.21.15.107:0/2342812899 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f567c001b70 con 0x7f568413ee40 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f5682ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f5682ffd640 1 -- 172.21.15.107:0/2342812899 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f567c001fc0 con 0x7f568413ee40 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f5682ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f5682ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f5682ffd640 10 monclient: dump: 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:33.393 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f568a8ed640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f568a8ed640 4 set_mon_vals no callback set 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f568a8ed640 10 monclient: shutdown 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f568a8ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f568a8ed640 1 -- 172.21.15.107:0/2342812899 >> v1:172.21.15.107:6789/0 conn(0x7f568413ee40 legacy=0x7f568413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f568a8ed640 1 -- 172.21.15.107:0/2342812899 shutdown_connections 2024-03-29T10:59:33.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.409+0000 7f568a8ed640 1 -- 172.21.15.107:0/2342812899 >> 172.21.15.107:0/2342812899 conn(0x7f568406b660 msgr2=0x7f568406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:33.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.410+0000 7f568a8ed640 1 -- 172.21.15.107:0/2342812899 shutdown_connections 2024-03-29T10:59:33.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.410+0000 7f568a8ed640 1 -- 172.21.15.107:0/2342812899 wait complete. 2024-03-29T10:59:33.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.410+0000 7f568a8ed640 10 monclient: build_initial_monmap 2024-03-29T10:59:33.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.410+0000 7f568a8ed640 10 monclient: monmap: 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.411+0000 7f568a8ed640 1 librados: starting msgr at 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.411+0000 7f568a8ed640 1 librados: starting objecter 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.411+0000 7f568a8ed640 1 Processor -- start 2024-03-29T10:59:33.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.411+0000 7f568a8ed640 1 -- start start 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.411+0000 7f568a8ed640 1 librados: setting wanted keys 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.411+0000 7f568a8ed640 1 librados: calling monclient init 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.411+0000 7f568a8ed640 10 monclient: init 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 10 monclient(hunting): picked mon.noname-a con 0x7f5684072450 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 10 monclient(hunting): picked mon.noname-c con 0x7f56840b0860 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 10 monclient(hunting): picked mon.noname-b con 0x7f568413ee40 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56840b1270 con 0x7f5684072450 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5684140d50 con 0x7f568413ee40 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5684143030 con 0x7f56840b0860 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f568a8ed640 10 monclient(hunting): authenticate will time out at 904.680664s 2024-03-29T10:59:33.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f56837fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f56840b0860 0x7f56840b1700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33782/0 (socket says 172.21.15.107:33782) 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.412+0000 7f56837fe640 1 -- 172.21.15.107:0/3893603319 learned_addr learned my addr 172.21.15.107:0/3893603319 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 client.?.objecter ms_handle_connect 0x7f56840b0860 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 client.?.objecter ms_handle_connect 0x7f5684072450 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 client.?.objecter ms_handle_connect 0x7f568413ee40 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486780643 0 0) 0x7f56840b1270 con 0x7f5684072450 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5670004eb0 con 0x7f5684072450 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512927278 0 0) 0x7f5684143030 con 0x7f56840b0860 2024-03-29T10:59:33.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.413+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56840b1270 con 0x7f56840b0860 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150575567 0 0) 0x7f5684140d50 con 0x7f568413ee40 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5684143030 con 0x7f568413ee40 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1668176051 0 0) 0x7f56840b1270 con 0x7f56840b0860 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5684140d50 con 0x7f56840b0860 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f5678001980 con 0x7f56840b0860 2024-03-29T10:59:33.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4177904950 0 0) 0x7f5684143030 con 0x7f568413ee40 2024-03-29T10:59:33.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56840b1270 con 0x7f568413ee40 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f567c001e80 con 0x7f568413ee40 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.414+0000 7f5680ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:33.408 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1572437663 0 0) 0x7f5670004eb0 con 0x7f5684072450 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5684143030 con 0x7f5684072450 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f56740018a0 con 0x7f5684072450 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:33.409 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3455651901 0 0) 0x7f56840b1270 con 0x7f568413ee40 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 >> v1:172.21.15.107:6790/0 conn(0x7f56840b0860 legacy=0x7f56840b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 >> v1:172.21.15.107:6789/0 conn(0x7f5684072450 legacy=0x7f5684138c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 1 monclient: found mon.b 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5684141ef0 con 0x7f568413ee40 2024-03-29T10:59:33.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 10 monclient: _finish_auth 0 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f5680ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f568a8ed640 5 monclient: authenticate success, global_id 4300 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f568a8ed640 10 monclient: _renew_subs 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f568a8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56840b4200 con 0x7f568413ee40 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f568a8ed640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f568a8ed640 10 monclient: _renew_subs 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f568a8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56841e7510 con 0x7f568413ee40 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.415+0000 7f568a8ed640 1 librados: init done 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f568a8ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:33.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f568a8ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f568a8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5684140f40 con 0x7f568413ee40 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f567c0012e0 con 0x7f568413ee40 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f5680ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f567c003c20 con 0x7f568413ee40 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f5680ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f5680ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f5680ff9640 10 monclient: dump: 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:33.412 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.417+0000 7f56827fc640 4 set_mon_vals no callback set 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.419+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f567c019bf0 con 0x7f568413ee40 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.420+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f567c083a40 con 0x7f568413ee40 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.420+0000 7f5680ff9640 10 client.4300.objecter ms_dispatch 0x7f5684000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.420+0000 7f5680ff9640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.420+0000 7f5680ff9640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.420+0000 7f5680ff9640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.421+0000 7f5683fff640 10 monclient: get_auth_request con 0x7f567006e1b0 auth_method 0 2024-03-29T10:59:33.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.421+0000 7f5680ff9640 10 client.4300.objecter ms_handle_connect 0x7f567006e1b0 2024-03-29T10:59:33.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.427+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f567c04f320 con 0x7f568413ee40 2024-03-29T10:59:33.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.427+0000 7f5680ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:33.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.427+0000 7f5680ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:33.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.784+0000 7f568a8ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:33.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.784+0000 7f568a8ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:33.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.784+0000 7f568a8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:33.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.784+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56840b0c40 con 0x7f568413ee40 2024-03-29T10:59:33.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.785+0000 7f5680ff9640 1 -- 172.21.15.107:0/3893603319 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f567c051250 con 0x7f568413ee40 2024-03-29T10:59:33.766 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.785+0000 7f5680ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:33.766 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.785+0000 7f5680ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:33.766 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:59:33.766 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.788+0000 7f568a8ed640 10 librados: watch_flush enter 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.788+0000 7f568a8ed640 10 librados: watch_flush exit 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.788+0000 7f568a8ed640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.788+0000 7f568a8ed640 10 client.4300.objecter successfully canceled tick 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.789+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 >> v1:172.21.15.107:6816/882194755 conn(0x7f567006e1b0 legacy=0x7f5670070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.789+0000 7f568a8ed640 10 monclient: shutdown 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.789+0000 7f568a8ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.789+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 >> v1:172.21.15.129:6789/0 conn(0x7f568413ee40 legacy=0x7f56841417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.789+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 shutdown_connections 2024-03-29T10:59:33.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.789+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 >> 172.21.15.107:0/3893603319 conn(0x7f568406b660 msgr2=0x7f56840a9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:33.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.789+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 shutdown_connections 2024-03-29T10:59:33.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.789+0000 7f568a8ed640 1 -- 172.21.15.107:0/3893603319 wait complete. 2024-03-29T10:59:33.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:33.790+0000 7f568a8ed640 1 librados: shutdown 2024-03-29T10:59:33.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:40.590 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:40.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.739+0000 7f5a020a8640 10 monclient: get_monmap_and_config 2024-03-29T10:59:40.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.739+0000 7f5a020a8640 10 monclient: build_initial_monmap 2024-03-29T10:59:40.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.739+0000 7f5a020a8640 10 monclient: monmap: 2024-03-29T10:59:40.719 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:40.719 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:40.719 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:59:40.740365+0000 2024-03-29T10:59:40.719 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:59:40.740365+0000 2024-03-29T10:59:40.720 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:40.720 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:40.720 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:40.720 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:59:40.720 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:59:40.720 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:40.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.740+0000 7f5a020a8640 1 Processor -- start 2024-03-29T10:59:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.740+0000 7f5a020a8640 1 -- start start 2024-03-29T10:59:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.740+0000 7f5a020a8640 10 monclient: init 2024-03-29T10:59:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.741+0000 7f5a020a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:40.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.741+0000 7f5a020a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:59:40.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.745+0000 7f5a020a8640 10 monclient(hunting): picked mon.noname-a con 0x7f59fc06c250 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:40.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.745+0000 7f59fb7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f59fc06c250 0x7f59fc1427b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47739/0 (socket says 172.21.15.107:47739) 2024-03-29T10:59:40.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.745+0000 7f59fb7fe640 1 -- 172.21.15.107:0/2197946532 learned_addr learned my addr 172.21.15.107:0/2197946532 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:40.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.745+0000 7f5a020a8640 10 monclient(hunting): picked mon.noname-c con 0x7f59fc0ae650 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:40.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f5a020a8640 10 monclient(hunting): picked mon.noname-b con 0x7f59fc13f8c0 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:40.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f5a020a8640 1 -- 172.21.15.107:0/2197946532 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc0adf50 con 0x7f59fc06c250 2024-03-29T10:59:40.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f5a020a8640 1 -- 172.21.15.107:0/2197946532 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc0ae230 con 0x7f59fc0ae650 2024-03-29T10:59:40.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f5a020a8640 1 -- 172.21.15.107:0/2197946532 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc13fd50 con 0x7f59fc13f8c0 2024-03-29T10:59:40.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f5a020a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:40.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f5a020a8640 10 monclient(hunting): authenticate will time out at 912.014621s 2024-03-29T10:59:40.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201812644 0 0) 0x7f59fc0adf50 con 0x7f59fc06c250 2024-03-29T10:59:40.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:59:40.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e4003520 con 0x7f59fc06c250 2024-03-29T10:59:40.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792490828 0 0) 0x7f59fc13fd50 con 0x7f59fc13f8c0 2024-03-29T10:59:40.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:59:40.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.746+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59fc0adf50 con 0x7f59fc13f8c0 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274270840 0 0) 0x7f59fc0ae230 con 0x7f59fc0ae650 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59fc13fd50 con 0x7f59fc0ae650 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 479159760 0 0) 0x7f59e4003520 con 0x7f59fc06c250 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 >> v1:172.21.15.107:6790/0 conn(0x7f59fc13f8c0 legacy=0x7f59fc14db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 >> v1:172.21.15.129:6789/0 conn(0x7f59fc0ae650 legacy=0x7f59fc13f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 monclient: found mon.noname-a 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T10:59:40.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc06db20 con 0x7f59fc06c250 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f59ec001690 con 0x7f59fc06c250 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: dump: 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.730 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f5a020a8640 5 monclient: authenticate success, global_id 4314 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59ec0017f0 con 0x7f59fc06c250 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f5a020a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f5a020a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 1 -- 172.21.15.107:0/2197946532 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f59ec001c40 con 0x7f59fc06c250 2024-03-29T10:59:40.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f5a020a8640 4 set_mon_vals no callback set 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.747+0000 7f59fa7fc640 10 monclient: dump: 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:40.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.748+0000 7f5a020a8640 10 monclient: shutdown 2024-03-29T10:59:40.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.748+0000 7f5a020a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:40.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.748+0000 7f5a020a8640 1 -- 172.21.15.107:0/2197946532 >> v1:172.21.15.107:6789/0 conn(0x7f59fc06c250 legacy=0x7f59fc1427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.748+0000 7f5a020a8640 1 -- 172.21.15.107:0/2197946532 shutdown_connections 2024-03-29T10:59:40.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.748+0000 7f5a020a8640 1 -- 172.21.15.107:0/2197946532 >> 172.21.15.107:0/2197946532 conn(0x7f59fc067610 msgr2=0x7f59fc0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:40.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.748+0000 7f5a020a8640 1 -- 172.21.15.107:0/2197946532 shutdown_connections 2024-03-29T10:59:40.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.748+0000 7f5a020a8640 1 -- 172.21.15.107:0/2197946532 wait complete. 2024-03-29T10:59:40.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.749+0000 7f5a020a8640 10 monclient: build_initial_monmap 2024-03-29T10:59:40.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.749+0000 7f5a020a8640 10 monclient: monmap: 2024-03-29T10:59:40.741 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.749+0000 7f5a020a8640 1 librados: starting msgr at 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.749+0000 7f5a020a8640 1 librados: starting objecter 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.749+0000 7f5a020a8640 1 Processor -- start 2024-03-29T10:59:40.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.750+0000 7f5a020a8640 1 -- start start 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.750+0000 7f5a020a8640 1 librados: setting wanted keys 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.750+0000 7f5a020a8640 1 librados: calling monclient init 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.750+0000 7f5a020a8640 10 monclient: init 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f5a020a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f5a020a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f5a020a8640 10 monclient(hunting): picked mon.noname-b con 0x7f59fc06c250 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f5a020a8640 10 monclient(hunting): picked mon.noname-c con 0x7f59fc0ae650 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f59fb7fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f59fc06c250 0x7f59fc070cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:46030/0 (socket says 172.21.15.107:46030) 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f59faffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f59fc0ae650 0x7f59fc1da410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49700/0 (socket says 172.21.15.107:49700) 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f5a020a8640 10 monclient(hunting): picked mon.noname-a con 0x7f59fc13f8c0 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f59fb7fe640 1 -- 172.21.15.107:0/2030840885 learned_addr learned my addr 172.21.15.107:0/2030840885 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f59e3fff640 10 client.?.objecter ms_handle_connect 0x7f59fc0ae650 2024-03-29T10:59:40.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f59e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f59e3fff640 10 client.?.objecter ms_handle_connect 0x7f59fc06c250 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f59e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc071890 con 0x7f59fc13f8c0 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc071400 con 0x7f59fc06c250 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.751+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc071590 con 0x7f59fc0ae650 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f5a020a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903002944 0 0) 0x7f59fc071400 con 0x7f59fc06c250 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f5a020a8640 10 monclient(hunting): authenticate will time out at 912.020330s 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:59:40.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e8004eb0 con 0x7f59fc06c250 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065429523 0 0) 0x7f59fc071590 con 0x7f59fc0ae650 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59fc071400 con 0x7f59fc0ae650 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3890306985 0 0) 0x7f59e8004eb0 con 0x7f59fc06c250 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59fc071590 con 0x7f59fc06c250 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f59ec001b00 con 0x7f59fc06c250 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): dump: 2024-03-29T10:59:40.745 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3402824398 0 0) 0x7f59fc071400 con 0x7f59fc0ae650 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59e8004eb0 con 0x7f59fc0ae650 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f59f00018b0 con 0x7f59fc0ae650 2024-03-29T10:59:40.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.752+0000 7f59e3fff640 10 monclient(hunting): dump: 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:40.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4284284214 0 0) 0x7f59fc071590 con 0x7f59fc06c250 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 >> v1:172.21.15.107:6790/0 conn(0x7f59fc0ae650 legacy=0x7f59fc1da410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 >> v1:172.21.15.107:6789/0 conn(0x7f59fc13f8c0 legacy=0x7f59fc1dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 1 monclient: found mon.b 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc1413d0 con 0x7f59fc06c250 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f59e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f5a020a8640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f5a020a8640 10 monclient: _renew_subs 2024-03-29T10:59:40.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f5a020a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.753+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc141620 con 0x7f59fc06c250 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f5a020a8640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f5a020a8640 10 monclient: _renew_subs 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f5a020a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc1e2c10 con 0x7f59fc06c250 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59ec004bf0 con 0x7f59fc06c250 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f5a020a8640 1 librados: init done 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f59e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f59f9ffb640 4 set_mon_vals no callback set 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f59ec004f00 con 0x7f59fc06c250 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f59e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:40.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f59e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.754+0000 7f59e3fff640 10 monclient: dump: 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.755+0000 7f5a020a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:40.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.755+0000 7f5a020a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.755+0000 7f5a020a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.755+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59fc075f10 con 0x7f59fc06c250 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.756+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f59ec01aed0 con 0x7f59fc06c250 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.756+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f59ec084a80 con 0x7f59fc06c250 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.756+0000 7f59e3fff640 10 client.4306.objecter ms_dispatch 0x7f59fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.756+0000 7f59e3fff640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.756+0000 7f59e3fff640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.757+0000 7f59e3fff640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.757+0000 7f59faffd640 10 monclient: get_auth_request con 0x7f59e806dde0 auth_method 0 2024-03-29T10:59:40.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.757+0000 7f59e3fff640 10 client.4306.objecter ms_handle_connect 0x7f59e806dde0 2024-03-29T10:59:40.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.762+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59ec050360 con 0x7f59fc06c250 2024-03-29T10:59:40.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.762+0000 7f59e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:40.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:40.762+0000 7f59e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:41.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.065+0000 7f5a020a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:41.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.065+0000 7f5a020a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:41.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.065+0000 7f5a020a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:41.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.065+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59fc074f50 con 0x7f59fc06c250 2024-03-29T10:59:41.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.065+0000 7f59e3fff640 1 -- 172.21.15.107:0/2030840885 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f59ec052290 con 0x7f59fc06c250 2024-03-29T10:59:41.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.065+0000 7f59e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:41.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.066+0000 7f59e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:41.047 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:59:41.047 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:41.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 10 librados: watch_flush enter 2024-03-29T10:59:41.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 10 librados: watch_flush exit 2024-03-29T10:59:41.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T10:59:41.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 10 client.4306.objecter successfully canceled tick 2024-03-29T10:59:41.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 >> v1:172.21.15.107:6816/882194755 conn(0x7f59e806dde0 legacy=0x7f59e80702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:41.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 10 monclient: shutdown 2024-03-29T10:59:41.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:41.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 >> v1:172.21.15.129:6789/0 conn(0x7f59fc06c250 legacy=0x7f59fc070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:41.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 shutdown_connections 2024-03-29T10:59:41.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 >> 172.21.15.107:0/2030840885 conn(0x7f59fc067610 msgr2=0x7f59fc13ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:41.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 shutdown_connections 2024-03-29T10:59:41.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.068+0000 7f5a020a8640 1 -- 172.21.15.107:0/2030840885 wait complete. 2024-03-29T10:59:41.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:41.069+0000 7f5a020a8640 1 librados: shutdown 2024-03-29T10:59:41.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:47.870 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:48.019 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.039+0000 7fb64610d640 10 monclient: get_monmap_and_config 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.039+0000 7fb64610d640 10 monclient: build_initial_monmap 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.039+0000 7fb64610d640 10 monclient: monmap: 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:59:48.040816+0000 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:59:48.040816+0000 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:59:48.020 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:48.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.041+0000 7fb64610d640 1 Processor -- start 2024-03-29T10:59:48.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.041+0000 7fb64610d640 1 -- start start 2024-03-29T10:59:48.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.041+0000 7fb64610d640 10 monclient: init 2024-03-29T10:59:48.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:48.022 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 10 monclient(hunting): picked mon.noname-c con 0x7fb6400b4890 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 10 monclient(hunting): picked mon.noname-b con 0x7fb64013c040 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 10 monclient(hunting): picked mon.noname-a con 0x7fb6401427e0 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400b42f0 con 0x7fb6401427e0 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400aa7a0 con 0x7fb6400b4890 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400aaa10 con 0x7fb64013c040 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.042+0000 7fb64610d640 10 monclient(hunting): authenticate will time out at 919.310309s 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.043+0000 7fb63f7fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb6400b4890 0x7fb64013b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:36734/0 (socket says 172.21.15.107:36734) 2024-03-29T10:59:48.023 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.043+0000 7fb63f7fe640 1 -- 172.21.15.107:0/3555865838 learned_addr learned my addr 172.21.15.107:0/3555865838 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:48.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 -- 172.21.15.107:0/3555865838 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917405655 0 0) 0x7fb6400aa7a0 con 0x7fb6400b4890 2024-03-29T10:59:48.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:48.028 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 -- 172.21.15.107:0/3555865838 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb628003520 con 0x7fb6400b4890 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 -- 172.21.15.107:0/3555865838 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 158946908 0 0) 0x7fb628003520 con 0x7fb6400b4890 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 -- 172.21.15.107:0/3555865838 >> v1:172.21.15.107:6790/0 conn(0x7fb64013c040 legacy=0x7fb6401420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 -- 172.21.15.107:0/3555865838 >> v1:172.21.15.107:6789/0 conn(0x7fb6401427e0 legacy=0x7fb64014d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 monclient: found mon.noname-c 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 -- 172.21.15.107:0/3555865838 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6400aacf0 con 0x7fb6400b4890 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:59:48.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 -- 172.21.15.107:0/3555865838 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb638001a10 con 0x7fb6400b4890 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: dump: 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:48.030 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb64610d640 5 monclient: authenticate success, global_id 4309 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb64610d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 -- 172.21.15.107:0/3555865838 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb638001b70 con 0x7fb6400b4890 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 1 -- 172.21.15.107:0/3555865838 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb638001f80 con 0x7fb6400b4890 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb63e7fc640 10 monclient: dump: 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.031 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.045+0000 7fb64610d640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.046+0000 7fb64610d640 4 set_mon_vals no callback set 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.047+0000 7fb64610d640 10 monclient: shutdown 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.047+0000 7fb64610d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.047+0000 7fb64610d640 1 -- 172.21.15.107:0/3555865838 >> v1:172.21.15.129:6789/0 conn(0x7fb6400b4890 legacy=0x7fb64013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.048+0000 7fb64610d640 1 -- 172.21.15.107:0/3555865838 shutdown_connections 2024-03-29T10:59:48.032 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.048+0000 7fb64610d640 1 -- 172.21.15.107:0/3555865838 >> 172.21.15.107:0/3555865838 conn(0x7fb64006d180 msgr2=0x7fb640066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:48.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.049+0000 7fb64610d640 1 -- 172.21.15.107:0/3555865838 shutdown_connections 2024-03-29T10:59:48.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.051+0000 7fb64610d640 1 -- 172.21.15.107:0/3555865838 wait complete. 2024-03-29T10:59:48.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.052+0000 7fb64610d640 10 monclient: build_initial_monmap 2024-03-29T10:59:48.036 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.052+0000 7fb64610d640 10 monclient: monmap: 2024-03-29T10:59:48.036 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:48.036 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:48.036 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:59:48.036 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:59:48.036 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:48.036 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:48.036 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.052+0000 7fb64610d640 1 librados: starting msgr at 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.052+0000 7fb64610d640 1 librados: starting objecter 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.053+0000 7fb64610d640 1 Processor -- start 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.053+0000 7fb64610d640 1 -- start start 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.053+0000 7fb64610d640 1 librados: setting wanted keys 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.053+0000 7fb64610d640 1 librados: calling monclient init 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.053+0000 7fb64610d640 10 monclient: init 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.054+0000 7fb64610d640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:48.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.054+0000 7fb64610d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:48.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb63f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb6400b4890 0x7fb640136050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54005/0 (socket says 172.21.15.107:54005) 2024-03-29T10:59:48.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb64610d640 10 monclient(hunting): picked mon.noname-a con 0x7fb6400b4890 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:48.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb64610d640 10 monclient(hunting): picked mon.noname-c con 0x7fb64013c040 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:48.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb64610d640 10 monclient(hunting): picked mon.noname-b con 0x7fb6401427e0 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:48.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb64610d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6400ad3c0 con 0x7fb6400b4890 2024-03-29T10:59:48.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb63f7fe640 1 -- 172.21.15.107:0/246137440 learned_addr learned my addr 172.21.15.107:0/246137440 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:48.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb640136760 con 0x7fb6401427e0 2024-03-29T10:59:48.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64013a9a0 con 0x7fb64013c040 2024-03-29T10:59:48.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb64610d640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:48.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.057+0000 7fb64610d640 10 monclient(hunting): authenticate will time out at 919.325788s 2024-03-29T10:59:48.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 client.?.objecter ms_handle_connect 0x7fb6400b4890 2024-03-29T10:59:48.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:48.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 client.?.objecter ms_handle_connect 0x7fb6401427e0 2024-03-29T10:59:48.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847017581 0 0) 0x7fb6400ad3c0 con 0x7fb6400b4890 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c004eb0 con 0x7fb6400b4890 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061804471 0 0) 0x7fb640136760 con 0x7fb6401427e0 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.058+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6400ad3c0 con 0x7fb6401427e0 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 440373360 0 0) 0x7fb62c004eb0 con 0x7fb6400b4890 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb640136760 con 0x7fb6400b4890 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb638001f00 con 0x7fb6400b4890 2024-03-29T10:59:48.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 10 monclient(hunting): dump: 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2728150097 0 0) 0x7fb6400ad3c0 con 0x7fb6401427e0 2024-03-29T10:59:48.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb62c004eb0 con 0x7fb6401427e0 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb630001830 con 0x7fb6401427e0 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 10 monclient(hunting): dump: 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3476202056 0 0) 0x7fb62c004eb0 con 0x7fb6401427e0 2024-03-29T10:59:48.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.059+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 >> v1:172.21.15.107:6790/0 conn(0x7fb64013c040 legacy=0x7fb64013a290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:48.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 >> v1:172.21.15.107:6789/0 conn(0x7fb6400b4890 legacy=0x7fb640136050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb627fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:48.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb627fff640 1 monclient: found mon.b 2024-03-29T10:59:48.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:48.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb627fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:48.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6400aef90 con 0x7fb6401427e0 2024-03-29T10:59:48.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb627fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:48.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb627fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:48.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb64610d640 5 monclient: authenticate success, global_id 4312 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb64610d640 10 monclient: _renew_subs 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb64610d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6400af270 con 0x7fb6401427e0 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb64610d640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb64610d640 10 monclient: _renew_subs 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb64610d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64013ebf0 con 0x7fb6401427e0 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.061+0000 7fb64610d640 1 librados: init done 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.062+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb630003610 con 0x7fb6401427e0 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.062+0000 7fb627fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.062+0000 7fb63dffb640 4 set_mon_vals no callback set 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.062+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb6300039a0 con 0x7fb6401427e0 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.062+0000 7fb627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.062+0000 7fb627fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:48.047 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.062+0000 7fb627fff640 10 monclient: dump: 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.063+0000 7fb64610d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.063+0000 7fb64610d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.063+0000 7fb64610d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.063+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb620005080 con 0x7fb6401427e0 2024-03-29T10:59:48.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.064+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb630003b00 con 0x7fb6401427e0 2024-03-29T10:59:48.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.065+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fb630083650 con 0x7fb6401427e0 2024-03-29T10:59:48.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.065+0000 7fb627fff640 10 client.4312.objecter ms_dispatch 0x7fb640000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:48.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.065+0000 7fb627fff640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:48.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.065+0000 7fb627fff640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:48.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.065+0000 7fb627fff640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T10:59:48.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.066+0000 7fb63f7fe640 10 monclient: get_auth_request con 0x7fb62c06df30 auth_method 0 2024-03-29T10:59:48.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.066+0000 7fb627fff640 10 client.4312.objecter ms_handle_connect 0x7fb62c06df30 2024-03-29T10:59:48.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.071+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb63004ef30 con 0x7fb6401427e0 2024-03-29T10:59:48.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.071+0000 7fb627fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:48.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.071+0000 7fb627fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:48.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.391+0000 7fb64610d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.391+0000 7fb64610d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.391+0000 7fb64610d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T10:59:48.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.391+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb620005330 con 0x7fb6401427e0 2024-03-29T10:59:48.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.392+0000 7fb63ffff640 1 -- 172.21.15.107:0/246137440 >> v1:172.21.15.129:6789/0 conn(0x7fb6401427e0 legacy=0x7fb6401f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:59:48.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.392+0000 7fb63ffff640 1 -- 172.21.15.107:0/246137440 >> v1:172.21.15.129:6789/0 conn(0x7fb6401427e0 legacy=0x7fb6401f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:59:48.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.392+0000 7fb63ffff640 1 --1- 172.21.15.107:0/246137440 >> v1:172.21.15.129:6789/0 conn(0x7fb6401427e0 0x7fb6401f3340 :-1 s=OPENED pgs=332276 cs=1 l=1).handle_message read tag failed 2024-03-29T10:59:48.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.392+0000 7fb63ffff640 1 --1- 172.21.15.107:0/246137440 >> v1:172.21.15.129:6789/0 conn(0x7fb6401427e0 0x7fb6401f3340 :-1 s=OPENED pgs=332276 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:59:48.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.392+0000 7fb627fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.129:6789/0 2024-03-29T10:59:48.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.392+0000 7fb627fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:48.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.392+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 >> v1:172.21.15.129:6789/0 conn(0x7fb6401427e0 legacy=0x7fb6401f3340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:59:48.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.392+0000 7fb627fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:48.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.393+0000 7fb627fff640 10 monclient(hunting): picked mon.a con 0x7fb6400b4890 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:48.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.393+0000 7fb627fff640 10 monclient(hunting): picked mon.c con 0x7fb64013c040 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:48.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.393+0000 7fb627fff640 10 monclient(hunting): picked mon.b con 0x7fb62c07a5f0 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:48.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.393+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb62c004eb0 con 0x7fb6400b4890 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.393+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb640136760 con 0x7fb62c07a5f0 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.393+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb64013a9a0 con 0x7fb64013c040 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.393+0000 7fb627fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 client.4312.objecter ms_handle_connect 0x7fb62c07a5f0 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 client.4312.objecter resend_mon_ops 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 client.4312.objecter ms_handle_connect 0x7fb64013c040 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 client.4312.objecter resend_mon_ops 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238817205 0 0) 0x7fb64013a9a0 con 0x7fb64013c040 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.107:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fb62c080730 con 0x7fb64013c040 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140509886 0 0) 0x7fb640136760 con 0x7fb62c07a5f0 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:48.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:48.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:59:48.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.129:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fb64013a9a0 con 0x7fb62c07a5f0 2024-03-29T10:59:48.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1386436901 0 0) 0x7fb62c080730 con 0x7fb64013c040 2024-03-29T10:59:48.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 >> v1:172.21.15.129:6789/0 conn(0x7fb62c07a5f0 legacy=0x7fb62c07ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.394+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 >> v1:172.21.15.107:6789/0 conn(0x7fb6400b4890 legacy=0x7fb62c074fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:48.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:48.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 1 monclient: found mon.c 2024-03-29T10:59:48.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:48.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fb6400ad3c0 con 0x7fb64013c040 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb62c004eb0 con 0x7fb64013c040 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb63004ef30 con 0x7fb64013c040 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: dump: 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.377 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:48.378 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:48.378 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:48.378 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:48.378 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:48.378 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:48.378 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:48.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb630053420 con 0x7fb64013c040 2024-03-29T10:59:48.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb627fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:48.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.395+0000 7fb63dffb640 4 set_mon_vals no callback set 2024-03-29T10:59:48.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.397+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb63004e9e0 con 0x7fb64013c040 2024-03-29T10:59:48.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.398+0000 7fb627fff640 1 -- 172.21.15.107:0/246137440 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fb630087050 con 0x7fb64013c040 2024-03-29T10:59:48.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.398+0000 7fb627fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:48.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.398+0000 7fb627fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:48.379 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:59:48.379 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:48.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 10 librados: watch_flush enter 2024-03-29T10:59:48.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 10 librados: watch_flush exit 2024-03-29T10:59:48.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T10:59:48.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 10 client.4312.objecter successfully canceled tick 2024-03-29T10:59:48.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 >> v1:172.21.15.107:6816/882194755 conn(0x7fb62c06df30 legacy=0x7fb62c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 10 monclient: shutdown 2024-03-29T10:59:48.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:48.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 >> v1:172.21.15.107:6790/0 conn(0x7fb64013c040 legacy=0x7fb62c078a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:48.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 shutdown_connections 2024-03-29T10:59:48.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.401+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 >> 172.21.15.107:0/246137440 conn(0x7fb64006d180 msgr2=0x7fb6400b5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:48.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.402+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 shutdown_connections 2024-03-29T10:59:48.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.402+0000 7fb64610d640 1 -- 172.21.15.107:0/246137440 wait complete. 2024-03-29T10:59:48.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:48.402+0000 7fb64610d640 1 librados: shutdown 2024-03-29T10:59:48.398 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:55.204 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:55.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.369+0000 7f451759e640 10 monclient: get_monmap_and_config 2024-03-29T10:59:55.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.369+0000 7f451759e640 10 monclient: build_initial_monmap 2024-03-29T10:59:55.349 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.369+0000 7f451759e640 10 monclient: monmap: 2024-03-29T10:59:55.349 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:55.350 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:55.350 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:59:55.370332+0000 2024-03-29T10:59:55.350 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:59:55.370332+0000 2024-03-29T10:59:55.350 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:55.350 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:55.350 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:55.350 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T10:59:55.350 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T10:59:55.350 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:55.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.370+0000 7f451759e640 1 Processor -- start 2024-03-29T10:59:55.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.370+0000 7f451759e640 1 -- start start 2024-03-29T10:59:55.351 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.370+0000 7f451759e640 10 monclient: init 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.371+0000 7f451759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.372+0000 7f451759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.372+0000 7f451759e640 10 monclient(hunting): picked mon.noname-a con 0x7f45100389e0 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.372+0000 7f451759e640 10 monclient(hunting): picked mon.noname-b con 0x7f4510038dc0 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.372+0000 7f451759e640 10 monclient(hunting): picked mon.noname-c con 0x7f4510030660 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.372+0000 7f451759e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100c54c0 con 0x7f45100389e0 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.372+0000 7f451759e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510036430 con 0x7f4510030660 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.372+0000 7f451759e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4510039840 con 0x7f4510038dc0 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.372+0000 7f451759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.372+0000 7f451759e640 10 monclient(hunting): authenticate will time out at 926.640548s 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.373+0000 7f4516d9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f4510030660 0x7f45100d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:47606/0 (socket says 172.21.15.107:47606) 2024-03-29T10:59:55.353 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.373+0000 7f4516d9d640 1 -- 172.21.15.107:0/3982464041 learned_addr learned my addr 172.21.15.107:0/3982464041 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:55.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.374+0000 7f451559a640 1 -- 172.21.15.107:0/3982464041 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839440506 0 0) 0x7f4510036430 con 0x7f4510030660 2024-03-29T10:59:55.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.374+0000 7f451559a640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.374+0000 7f451559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.374+0000 7f451559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.374+0000 7f451559a640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:59:55.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.374+0000 7f451559a640 1 -- 172.21.15.107:0/3982464041 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4500003520 con 0x7f4510030660 2024-03-29T10:59:55.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.375+0000 7f451559a640 1 -- 172.21.15.107:0/3982464041 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2241932290 0 0) 0x7f4500003520 con 0x7f4510030660 2024-03-29T10:59:55.355 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.375+0000 7f451559a640 1 -- 172.21.15.107:0/3982464041 >> v1:172.21.15.107:6790/0 conn(0x7f4510038dc0 legacy=0x7f45100cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:55.356 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 1 -- 172.21.15.107:0/3982464041 >> v1:172.21.15.107:6789/0 conn(0x7f45100389e0 legacy=0x7f4510033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 10 monclient: _finish_hunting 0 2024-03-29T10:59:55.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 1 monclient: found mon.noname-c 2024-03-29T10:59:55.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:55.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T10:59:55.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 1 -- 172.21.15.107:0/3982464041 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4510035690 con 0x7f4510030660 2024-03-29T10:59:55.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 10 monclient: _finish_auth 0 2024-03-29T10:59:55.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 10 monclient: _check_auth_tickets 2024-03-29T10:59:55.357 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 1 -- 172.21.15.107:0/3982464041 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f451804f730 con 0x7f4510030660 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451559a640 10 monclient: dump: 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:55.358 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:55.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451759e640 5 monclient: authenticate success, global_id 4315 2024-03-29T10:59:55.359 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.376+0000 7f451759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:55.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.379+0000 7f451559a640 1 -- 172.21.15.107:0/3982464041 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4518075f10 con 0x7f4510030660 2024-03-29T10:59:55.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.379+0000 7f451559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:55.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.379+0000 7f451559a640 1 -- 172.21.15.107:0/3982464041 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f45180758b0 con 0x7f4510030660 2024-03-29T10:59:55.360 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.379+0000 7f451559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.379+0000 7f451559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.379+0000 7f451559a640 10 monclient: dump: 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.379+0000 7f451759e640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.379+0000 7f451759e640 4 set_mon_vals no callback set 2024-03-29T10:59:55.361 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.380+0000 7f451759e640 10 monclient: shutdown 2024-03-29T10:59:55.362 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.380+0000 7f451759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:55.362 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.380+0000 7f451759e640 1 -- 172.21.15.107:0/3982464041 >> v1:172.21.15.129:6789/0 conn(0x7f4510030660 legacy=0x7f45100d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.381+0000 7f451759e640 1 -- 172.21.15.107:0/3982464041 shutdown_connections 2024-03-29T10:59:55.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.381+0000 7f451759e640 1 -- 172.21.15.107:0/3982464041 >> 172.21.15.107:0/3982464041 conn(0x7f4510005c00 msgr2=0x7f4510006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:55.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.381+0000 7f451759e640 1 -- 172.21.15.107:0/3982464041 shutdown_connections 2024-03-29T10:59:55.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.381+0000 7f451759e640 1 -- 172.21.15.107:0/3982464041 wait complete. 2024-03-29T10:59:55.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.382+0000 7f451759e640 10 monclient: build_initial_monmap 2024-03-29T10:59:55.363 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.382+0000 7f451759e640 10 monclient: monmap: 2024-03-29T10:59:55.363 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.383+0000 7f451759e640 1 librados: starting msgr at 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.383+0000 7f451759e640 1 librados: starting objecter 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.383+0000 7f451759e640 1 Processor -- start 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.383+0000 7f451759e640 1 -- start start 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.383+0000 7f451759e640 1 librados: setting wanted keys 2024-03-29T10:59:55.364 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.383+0000 7f451759e640 1 librados: calling monclient init 2024-03-29T10:59:55.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.383+0000 7f451759e640 10 monclient: init 2024-03-29T10:59:55.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.384+0000 7f451759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:55.365 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.384+0000 7f451759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:55.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.386+0000 7f451759e640 10 monclient(hunting): picked mon.noname-c con 0x7f4510030660 addr v1:172.21.15.107:6790/0 2024-03-29T10:59:55.366 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.387+0000 7f451759e640 10 monclient(hunting): picked mon.noname-a con 0x7f45100389e0 addr v1:172.21.15.107:6789/0 2024-03-29T10:59:55.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.387+0000 7f451759e640 10 monclient(hunting): picked mon.noname-b con 0x7f45100c7db0 addr v1:172.21.15.129:6789/0 2024-03-29T10:59:55.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.387+0000 7f451659c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4510030660 0x7f451002f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45793/0 (socket says 172.21.15.107:45793) 2024-03-29T10:59:55.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.387+0000 7f451659c640 1 -- 172.21.15.107:0/3785316066 learned_addr learned my addr 172.21.15.107:0/3785316066 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T10:59:55.367 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.387+0000 7f4516d9d640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f45100c7db0 0x7f45101641e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:48970/0 (socket says 172.21.15.107:48970) 2024-03-29T10:59:55.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451002fd90 con 0x7f45100389e0 2024-03-29T10:59:55.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451000a2c0 con 0x7f45100c7db0 2024-03-29T10:59:55.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451000a510 con 0x7f4510030660 2024-03-29T10:59:55.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f451759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:55.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f451759e640 10 monclient(hunting): authenticate will time out at 926.656255s 2024-03-29T10:59:55.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f4506ffd640 10 client.?.objecter ms_handle_connect 0x7f4510030660 2024-03-29T10:59:55.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f4506ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:55.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999925449 0 0) 0x7f451000a510 con 0x7f4510030660 2024-03-29T10:59:55.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f4506ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:55.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f4506ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:55.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f4506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:55.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f4506ffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:59:55.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f8004eb0 con 0x7f4510030660 2024-03-29T10:59:55.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.388+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3903636474 0 0) 0x7f44f8004eb0 con 0x7f4510030660 2024-03-29T10:59:55.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f451000a510 con 0x7f4510030660 2024-03-29T10:59:55.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4508001930 con 0x7f4510030660 2024-03-29T10:59:55.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:55.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 10 monclient(hunting): dump: 2024-03-29T10:59:55.374 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:55.374 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:55.374 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:55.374 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:55.374 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.375 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:55.375 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:55.375 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:55.375 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:55.375 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:55.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:55.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 10 client.?.objecter ms_handle_connect 0x7f45100c7db0 2024-03-29T10:59:55.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:55.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3562064236 0 0) 0x7f451000a510 con 0x7f4510030660 2024-03-29T10:59:55.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 >> v1:172.21.15.129:6789/0 conn(0x7f45100c7db0 legacy=0x7f45101641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.389+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 >> v1:172.21.15.107:6789/0 conn(0x7f45100389e0 legacy=0x7f451000cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.390+0000 7f4506ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:55.377 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.390+0000 7f4506ffd640 1 monclient: found mon.c 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.390+0000 7f4506ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.390+0000 7f4506ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.390+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451000d670 con 0x7f4510030660 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f4506ffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f4506ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f4506ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f4506ffd640 10 client.?.objecter ms_handle_connect 0x7f45100389e0 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f4506ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f451759e640 5 monclient: authenticate success, global_id 4331 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f451759e640 10 monclient: _renew_subs 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f451759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451000d8c0 con 0x7f4510030660 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f451759e640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:55.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f451759e640 10 monclient: _renew_subs 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f451759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4510164ad0 con 0x7f4510030660 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.391+0000 7f451759e640 1 librados: init done 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.392+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4508001d60 con 0x7f4510030660 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.392+0000 7f4506ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.392+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4508003a60 con 0x7f4510030660 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.392+0000 7f4506ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.392+0000 7f4506ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.392+0000 7f4506ffd640 10 monclient: dump: 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:55.379 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.392+0000 7f4507fff640 4 set_mon_vals no callback set 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.393+0000 7f451759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.393+0000 7f451759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.393+0000 7f451759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.393+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc005080 con 0x7f4510030660 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.394+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f4508003bc0 con 0x7f4510030660 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.395+0000 7f4515d9b640 10 monclient: get_auth_request con 0x7f44f806dc00 auth_method 0 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.395+0000 7f4506ffd640 10 client.4331.objecter ms_handle_connect 0x7f44f806dc00 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.395+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f4508084b20 con 0x7f4510030660 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.395+0000 7f4506ffd640 10 client.4331.objecter ms_dispatch 0x7f4518000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.395+0000 7f4506ffd640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:55.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.395+0000 7f4506ffd640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:55.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.395+0000 7f4506ffd640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T10:59:55.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.403+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4508050230 con 0x7f4510030660 2024-03-29T10:59:55.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.403+0000 7f4506ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:55.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.403+0000 7f4506ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:55.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.719+0000 7f451759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:55.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.719+0000 7f451759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:55.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.719+0000 7f451759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T10:59:55.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.719+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44fc005330 con 0x7f4510030660 2024-03-29T10:59:55.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.719+0000 7f4506ffd640 1 -- 172.21.15.107:0/3785316066 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f4508052330 con 0x7f4510030660 2024-03-29T10:59:55.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.719+0000 7f4506ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:55.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.719+0000 7f4506ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:55.700 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T10:59:55.700 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.723+0000 7f451759e640 10 librados: watch_flush enter 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.723+0000 7f451759e640 10 librados: watch_flush exit 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.723+0000 7f451759e640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.723+0000 7f451759e640 10 client.4331.objecter successfully canceled tick 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.724+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 >> v1:172.21.15.107:6816/882194755 conn(0x7f44f806dc00 legacy=0x7f44f80700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.724+0000 7f451759e640 10 monclient: shutdown 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.724+0000 7f451759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.724+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 >> v1:172.21.15.107:6790/0 conn(0x7f4510030660 legacy=0x7f451002f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.724+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 shutdown_connections 2024-03-29T10:59:55.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.724+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 >> 172.21.15.107:0/3785316066 conn(0x7f4510005c00 msgr2=0x7f451003ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:55.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.725+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 shutdown_connections 2024-03-29T10:59:55.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.725+0000 7f451759e640 1 -- 172.21.15.107:0/3785316066 wait complete. 2024-03-29T10:59:55.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T10:59:55.725+0000 7f451759e640 1 librados: shutdown 2024-03-29T10:59:55.725 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:02.532 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:02.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.696+0000 7fe20f74d640 10 monclient: get_monmap_and_config 2024-03-29T11:00:02.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.696+0000 7fe20f74d640 10 monclient: build_initial_monmap 2024-03-29T11:00:02.676 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.696+0000 7fe20f74d640 10 monclient: monmap: 2024-03-29T11:00:02.676 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:02.676 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:02.677 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:00:02.697288+0000 2024-03-29T11:00:02.677 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:00:02.697288+0000 2024-03-29T11:00:02.677 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:02.677 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:02.677 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:02.677 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:00:02.677 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:00:02.677 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:02.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.697+0000 7fe20f74d640 1 Processor -- start 2024-03-29T11:00:02.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.697+0000 7fe20f74d640 1 -- start start 2024-03-29T11:00:02.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.697+0000 7fe20f74d640 10 monclient: init 2024-03-29T11:00:02.678 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.698+0000 7fe20f74d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:02.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.698+0000 7fe20f74d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:02.679 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.698+0000 7fe20f74d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2080b4890 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:02.682 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.702+0000 7fe20f74d640 10 monclient(hunting): picked mon.noname-c con 0x7fe20813c040 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:02.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.702+0000 7fe20f74d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2081427e0 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:02.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.702+0000 7fe20cce8640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fe20813c040 0x7fe2081420d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:50753/0 (socket says 172.21.15.107:50753) 2024-03-29T11:00:02.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.705+0000 7fe20cce8640 1 -- 172.21.15.107:0/236851860 learned_addr learned my addr 172.21.15.107:0/236851860 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:02.694 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.705+0000 7fe20f74d640 1 -- 172.21.15.107:0/236851860 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2080b42f0 con 0x7fe2080b4890 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe20f74d640 1 -- 172.21.15.107:0/236851860 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2080aa7a0 con 0x7fe20813c040 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.705+0000 7fe20d4e9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe2080b4890 0x7fe20813b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54497/0 (socket says 172.21.15.107:54497) 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.705+0000 7fe20dcea640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fe2081427e0 0x7fe20814d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50061/0 (socket says 172.21.15.107:50061) 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe20f74d640 1 -- 172.21.15.107:0/236851860 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2080aaa10 con 0x7fe2081427e0 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 1 -- 172.21.15.107:0/236851860 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929187479 0 0) 0x7fe2080aa7a0 con 0x7fe20813c040 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe20f74d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe20f74d640 10 monclient(hunting): authenticate will time out at 933.974501s 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 1 -- 172.21.15.107:0/236851860 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f4003520 con 0x7fe20813c040 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 1 -- 172.21.15.107:0/236851860 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1097551707 0 0) 0x7fe1f4003520 con 0x7fe20813c040 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 1 -- 172.21.15.107:0/236851860 >> v1:172.21.15.107:6790/0 conn(0x7fe2081427e0 legacy=0x7fe20814d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:02.695 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 1 -- 172.21.15.107:0/236851860 >> v1:172.21.15.107:6789/0 conn(0x7fe2080b4890 legacy=0x7fe20813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 1 monclient: found mon.noname-c 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 1 -- 172.21.15.107:0/236851860 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2080aacf0 con 0x7fe20813c040 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient: _finish_auth 0 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 1 -- 172.21.15.107:0/236851860 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fe200001a10 con 0x7fe20813c040 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.706+0000 7fe1fffff640 10 monclient: dump: 2024-03-29T11:00:02.696 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.707+0000 7fe1fffff640 1 -- 172.21.15.107:0/236851860 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe200001b70 con 0x7fe20813c040 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.707+0000 7fe1fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.707+0000 7fe1fffff640 1 -- 172.21.15.107:0/236851860 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fe200001f80 con 0x7fe20813c040 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.707+0000 7fe1fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.697 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.707+0000 7fe1fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.707+0000 7fe1fffff640 10 monclient: dump: 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.707+0000 7fe20f74d640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.707+0000 7fe20f74d640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:02.698 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.707+0000 7fe20f74d640 4 set_mon_vals no callback set 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.708+0000 7fe20f74d640 10 monclient: shutdown 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.708+0000 7fe20f74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.708+0000 7fe20f74d640 1 -- 172.21.15.107:0/236851860 >> v1:172.21.15.129:6789/0 conn(0x7fe20813c040 legacy=0x7fe2081420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.709+0000 7fe20f74d640 1 -- 172.21.15.107:0/236851860 shutdown_connections 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.709+0000 7fe20f74d640 1 -- 172.21.15.107:0/236851860 >> 172.21.15.107:0/236851860 conn(0x7fe20806d180 msgr2=0x7fe208066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.709+0000 7fe20f74d640 1 -- 172.21.15.107:0/236851860 shutdown_connections 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.709+0000 7fe20f74d640 1 -- 172.21.15.107:0/236851860 wait complete. 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.710+0000 7fe20f74d640 10 monclient: build_initial_monmap 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.710+0000 7fe20f74d640 10 monclient: monmap: 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:02.699 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.711+0000 7fe20f74d640 1 librados: starting msgr at 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.711+0000 7fe20f74d640 1 librados: starting objecter 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.711+0000 7fe20f74d640 1 Processor -- start 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.711+0000 7fe20f74d640 1 -- start start 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.711+0000 7fe20f74d640 1 librados: setting wanted keys 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.711+0000 7fe20f74d640 1 librados: calling monclient init 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.711+0000 7fe20f74d640 10 monclient: init 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 10 monclient(hunting): picked mon.noname-c con 0x7fe2080b4890 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:02.700 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 10 monclient(hunting): picked mon.noname-a con 0x7fe20813c040 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:02.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2081427e0 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:02.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe208146390 con 0x7fe20813c040 2024-03-29T11:00:02.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2080ac7c0 con 0x7fe2081427e0 2024-03-29T11:00:02.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2080aca30 con 0x7fe2080b4890 2024-03-29T11:00:02.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:02.701 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.713+0000 7fe20f74d640 10 monclient(hunting): authenticate will time out at 933.981487s 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe20dcea640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fe2081427e0 0x7fe2081e17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:50807/0 (socket says 172.21.15.107:50807) 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe20dcea640 1 -- 172.21.15.107:0/3074637562 learned_addr learned my addr 172.21.15.107:0/3074637562 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 10 client.?.objecter ms_handle_connect 0x7fe2081427e0 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 10 client.?.objecter ms_handle_connect 0x7fe2080b4890 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087456867 0 0) 0x7fe2080ac7c0 con 0x7fe2081427e0 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:00:02.704 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.717+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1ec004eb0 con 0x7fe2081427e0 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 client.?.objecter ms_handle_connect 0x7fe20813c040 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911632 0 0) 0x7fe2080aca30 con 0x7fe2080b4890 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2080ac7c0 con 0x7fe2080b4890 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 848596219 0 0) 0x7fe1ec004eb0 con 0x7fe2081427e0 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2080aca30 con 0x7fe2081427e0 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 732377734 0 0) 0x7fe2080ac7c0 con 0x7fe2080b4890 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1ec004eb0 con 0x7fe2080b4890 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fe204001510 con 0x7fe2081427e0 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.705 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): dump: 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fe1f8001810 con 0x7fe2080b4890 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.706 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 10 monclient(hunting): dump: 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.718+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4000436202 0 0) 0x7fe1ec004eb0 con 0x7fe2080b4890 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 >> v1:172.21.15.129:6789/0 conn(0x7fe2081427e0 legacy=0x7fe2081e17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.707 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 >> v1:172.21.15.107:6789/0 conn(0x7fe20813c040 legacy=0x7fe2081d7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 1 monclient: found mon.c 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2080ab900 con 0x7fe2080b4890 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe1fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe20f74d640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe20f74d640 10 monclient: _renew_subs 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe20f74d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2080abaf0 con 0x7fe2080b4890 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe20f74d640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe20f74d640 10 monclient: _renew_subs 2024-03-29T11:00:02.708 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe20f74d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2080ab6e0 con 0x7fe2080b4890 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.719+0000 7fe20f74d640 1 librados: init done 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.721+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1f8001d40 con 0x7fe2080b4890 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.721+0000 7fe1fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.721+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fe1f8004980 con 0x7fe2080b4890 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.721+0000 7fe1fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.721+0000 7fe1fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.721+0000 7fe1fdffb640 10 monclient: dump: 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:02.709 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.721+0000 7fe1feffd640 4 set_mon_vals no callback set 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.722+0000 7fe20f74d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.722+0000 7fe20f74d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.722+0000 7fe20f74d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.722+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e8005080 con 0x7fe2080b4890 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.722+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fe1f8004ba0 con 0x7fe2080b4890 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.723+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fe1f80847c0 con 0x7fe2080b4890 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.723+0000 7fe1fdffb640 10 client.4334.objecter ms_dispatch 0x7fe208000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:02.710 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.723+0000 7fe1fdffb640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:02.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.723+0000 7fe1fdffb640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:02.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.723+0000 7fe1fdffb640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T11:00:02.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.723+0000 7fe20cce8640 10 monclient: get_auth_request con 0x7fe1ec06ded0 auth_method 0 2024-03-29T11:00:02.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.725+0000 7fe1fdffb640 10 client.4334.objecter ms_handle_connect 0x7fe1ec06ded0 2024-03-29T11:00:02.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.730+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1f80500a0 con 0x7fe2080b4890 2024-03-29T11:00:02.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.730+0000 7fe1fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:02.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:02.730+0000 7fe1fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:03.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.054+0000 7fe20f74d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:03.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.054+0000 7fe20f74d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:03.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.054+0000 7fe20f74d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:03.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.054+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1e8005330 con 0x7fe2080b4890 2024-03-29T11:00:03.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.055+0000 7fe1fdffb640 1 -- 172.21.15.107:0/3074637562 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fe1f8051fd0 con 0x7fe2080b4890 2024-03-29T11:00:03.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.055+0000 7fe1fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:03.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.055+0000 7fe1fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:03.038 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:00:03.038 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:03.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.061+0000 7fe20f74d640 10 librados: watch_flush enter 2024-03-29T11:00:03.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.064+0000 7fe20f74d640 10 librados: watch_flush exit 2024-03-29T11:00:03.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.064+0000 7fe20f74d640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T11:00:03.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.064+0000 7fe20f74d640 10 client.4334.objecter successfully canceled tick 2024-03-29T11:00:03.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.065+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 >> v1:172.21.15.107:6816/882194755 conn(0x7fe1ec06ded0 legacy=0x7fe1ec070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:03.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.065+0000 7fe20f74d640 10 monclient: shutdown 2024-03-29T11:00:03.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.065+0000 7fe20f74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:03.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.065+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 >> v1:172.21.15.107:6790/0 conn(0x7fe2080b4890 legacy=0x7fe2081d6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:03.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.065+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 shutdown_connections 2024-03-29T11:00:03.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.065+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 >> 172.21.15.107:0/3074637562 conn(0x7fe20806d180 msgr2=0x7fe2080b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:03.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.065+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 shutdown_connections 2024-03-29T11:00:03.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.065+0000 7fe20f74d640 1 -- 172.21.15.107:0/3074637562 wait complete. 2024-03-29T11:00:03.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:03.066+0000 7fe20f74d640 1 librados: shutdown 2024-03-29T11:00:03.065 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:09.872 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:10.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.050+0000 7fc2cba74640 10 monclient: get_monmap_and_config 2024-03-29T11:00:10.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.050+0000 7fc2cba74640 10 monclient: build_initial_monmap 2024-03-29T11:00:10.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.050+0000 7fc2cba74640 10 monclient: monmap: 2024-03-29T11:00:10.030 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:10.030 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:10.030 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:00:10.051029+0000 2024-03-29T11:00:10.030 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:00:10.051029+0000 2024-03-29T11:00:10.030 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:10.031 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:10.031 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:10.031 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:00:10.031 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:00:10.031 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:10.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.050+0000 7fc2cba74640 1 Processor -- start 2024-03-29T11:00:10.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.051+0000 7fc2cba74640 1 -- start start 2024-03-29T11:00:10.031 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.051+0000 7fc2cba74640 10 monclient: init 2024-03-29T11:00:10.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.051+0000 7fc2cba74640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:10.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.052+0000 7fc2cba74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:10.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.052+0000 7fc2cba74640 10 monclient(hunting): picked mon.noname-a con 0x7fc2c40b0770 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:10.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.052+0000 7fc2cba74640 10 monclient(hunting): picked mon.noname-c con 0x7fc2c414a700 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:10.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.052+0000 7fc2cba74640 10 monclient(hunting): picked mon.noname-b con 0x7fc2c40ac5a0 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:10.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.052+0000 7fc2cba74640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c40aba70 con 0x7fc2c40b0770 2024-03-29T11:00:10.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.052+0000 7fc2cba74640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c40aecb0 con 0x7fc2c414a700 2024-03-29T11:00:10.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.052+0000 7fc2cba74640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c40af150 con 0x7fc2c40ac5a0 2024-03-29T11:00:10.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.052+0000 7fc2cba74640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:10.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.052+0000 7fc2cba74640 10 monclient(hunting): authenticate will time out at 941.320072s 2024-03-29T11:00:10.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.054+0000 7fc2c900f640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc2c414a700 0x7fc2c40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:40321/0 (socket says 172.21.15.107:40321) 2024-03-29T11:00:10.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.054+0000 7fc2c900f640 1 -- 172.21.15.107:0/1715008171 learned_addr learned my addr 172.21.15.107:0/1715008171 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:10.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124231018 0 0) 0x7fc2c40aecb0 con 0x7fc2c414a700 2024-03-29T11:00:10.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 10 monclient(hunting): _negotiate 2024-03-29T11:00:10.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:10.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:10.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:00:10.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b0003520 con 0x7fc2c414a700 2024-03-29T11:00:10.037 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455239072 0 0) 0x7fc2c40aba70 con 0x7fc2c40b0770 2024-03-29T11:00:10.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 10 monclient(hunting): _negotiate 2024-03-29T11:00:10.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:10.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:10.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:00:10.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.055+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c40aecb0 con 0x7fc2c40b0770 2024-03-29T11:00:10.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.056+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 765451885 0 0) 0x7fc2c40aecb0 con 0x7fc2c40b0770 2024-03-29T11:00:10.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.056+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 >> v1:172.21.15.107:6790/0 conn(0x7fc2c40ac5a0 legacy=0x7fc2c4151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 >> v1:172.21.15.129:6789/0 conn(0x7fc2c414a700 legacy=0x7fc2c40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 10 monclient: _finish_hunting 0 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 1 monclient: found mon.noname-a 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c40ac9d0 con 0x7fc2c40b0770 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 10 monclient: _finish_auth 0 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 10 monclient: _check_auth_tickets 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:10.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2cba74640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2cba74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc2c0001670 con 0x7fc2c40b0770 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2c880e640 10 monclient: dump: 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:10.040 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:10.041 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:10.041 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:10.041 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:10.041 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:10.041 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:10.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.058+0000 7fc2cba74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:10.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.062+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2c00018c0 con 0x7fc2c40b0770 2024-03-29T11:00:10.042 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.062+0000 7fc2c880e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.062+0000 7fc2cba74640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.062+0000 7fc2cba74640 4 set_mon_vals no callback set 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.062+0000 7fc2c880e640 1 -- 172.21.15.107:0/1715008171 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc2c0001c50 con 0x7fc2c40b0770 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.062+0000 7fc2c880e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.062+0000 7fc2c880e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.062+0000 7fc2c880e640 10 monclient: dump: 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:10.043 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:10.044 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:10.044 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:10.044 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:10.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.063+0000 7fc2cba74640 10 monclient: shutdown 2024-03-29T11:00:10.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.063+0000 7fc2cba74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:10.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.063+0000 7fc2cba74640 1 -- 172.21.15.107:0/1715008171 >> v1:172.21.15.107:6789/0 conn(0x7fc2c40b0770 legacy=0x7fc2c4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.063+0000 7fc2cba74640 1 -- 172.21.15.107:0/1715008171 shutdown_connections 2024-03-29T11:00:10.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.063+0000 7fc2cba74640 1 -- 172.21.15.107:0/1715008171 >> 172.21.15.107:0/1715008171 conn(0x7fc2c406b660 msgr2=0x7fc2c406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:10.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.063+0000 7fc2cba74640 1 -- 172.21.15.107:0/1715008171 shutdown_connections 2024-03-29T11:00:10.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.064+0000 7fc2cba74640 1 -- 172.21.15.107:0/1715008171 wait complete. 2024-03-29T11:00:10.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.064+0000 7fc2cba74640 10 monclient: build_initial_monmap 2024-03-29T11:00:10.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.064+0000 7fc2cba74640 10 monclient: monmap: 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.065+0000 7fc2cba74640 1 librados: starting msgr at 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.065+0000 7fc2cba74640 1 librados: starting objecter 2024-03-29T11:00:10.051 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.065+0000 7fc2cba74640 1 Processor -- start 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.065+0000 7fc2cba74640 1 -- start start 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.065+0000 7fc2cba74640 1 librados: setting wanted keys 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.065+0000 7fc2cba74640 1 librados: calling monclient init 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.065+0000 7fc2cba74640 10 monclient: init 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 10 monclient(hunting): picked mon.noname-a con 0x7fc2c40ac5a0 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 10 monclient(hunting): picked mon.noname-b con 0x7fc2c40b0770 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 10 monclient(hunting): picked mon.noname-c con 0x7fc2c414a700 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c41e4250 con 0x7fc2c40ac5a0 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c41e4df0 con 0x7fc2c40b0770 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c411fb20 con 0x7fc2c414a700 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:10.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.066+0000 7fc2cba74640 10 monclient(hunting): authenticate will time out at 941.334287s 2024-03-29T11:00:10.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ca011640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc2c414a700 0x7fc2c41ee990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:59891/0 (socket says 172.21.15.107:59891) 2024-03-29T11:00:10.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ca011640 1 -- 172.21.15.107:0/443162650 learned_addr learned my addr 172.21.15.107:0/443162650 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:10.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ba7fc640 10 client.?.objecter ms_handle_connect 0x7fc2c40ac5a0 2024-03-29T11:00:10.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:10.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33538406 0 0) 0x7fc2c41e4250 con 0x7fc2c40ac5a0 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ba7fc640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.072+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a8004eb0 con 0x7fc2c40ac5a0 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 client.?.objecter ms_handle_connect 0x7fc2c414a700 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667208788 0 0) 0x7fc2c411fb20 con 0x7fc2c414a700 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c41e4250 con 0x7fc2c414a700 2024-03-29T11:00:10.055 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3625677450 0 0) 0x7fc2c41e4250 con 0x7fc2c414a700 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2c411fb20 con 0x7fc2c414a700 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc2bc004340 con 0x7fc2c414a700 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 monclient(hunting): dump: 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:10.056 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3878202157 0 0) 0x7fc2c411fb20 con 0x7fc2c414a700 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 >> v1:172.21.15.129:6789/0 conn(0x7fc2c40b0770 legacy=0x7fc2c411dea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 >> v1:172.21.15.107:6789/0 conn(0x7fc2c40ac5a0 legacy=0x7fc2c41e46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.073+0000 7fc2ba7fc640 1 monclient: found mon.c 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2ba7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c41e3ee0 con 0x7fc2c414a700 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2ba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2ba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2cba74640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:00:10.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2cba74640 10 monclient: _renew_subs 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2cba74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c411e5b0 con 0x7fc2c414a700 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2cba74640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2cba74640 10 monclient: _renew_subs 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2cba74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c411b1c0 con 0x7fc2c414a700 2024-03-29T11:00:10.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.074+0000 7fc2cba74640 1 librados: init done 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.075+0000 7fc2cba74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2cba74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2cba74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c4075f10 con 0x7fc2c414a700 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2bc004850 con 0x7fc2c414a700 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc2bc0013f0 con 0x7fc2c414a700 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2ba7fc640 10 monclient: dump: 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:10.059 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:10.060 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:10.060 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:10.060 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:10.060 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:10.060 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:10.060 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:10.060 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:10.060 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:10.060 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.076+0000 7fc2bbfff640 4 set_mon_vals no callback set 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.080+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fc2bc001630 con 0x7fc2c414a700 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.080+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fc2bc084cb0 con 0x7fc2c414a700 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.080+0000 7fc2ba7fc640 10 client.4337.objecter ms_dispatch 0x7fc2c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.080+0000 7fc2ba7fc640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.080+0000 7fc2ba7fc640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.080+0000 7fc2ba7fc640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.080+0000 7fc2c9810640 10 monclient: get_auth_request con 0x7fc2a806dd30 auth_method 0 2024-03-29T11:00:10.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.081+0000 7fc2ba7fc640 10 client.4337.objecter ms_handle_connect 0x7fc2a806dd30 2024-03-29T11:00:10.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.086+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2bc050590 con 0x7fc2c414a700 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.086+0000 7fc2ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:10.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.086+0000 7fc2ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:10.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.402+0000 7fc2cba74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:10.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.402+0000 7fc2cba74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:10.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.402+0000 7fc2cba74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:10.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.402+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2c40b2690 con 0x7fc2c414a700 2024-03-29T11:00:10.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.404+0000 7fc2ba7fc640 1 -- 172.21.15.107:0/443162650 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fc2bc0524c0 con 0x7fc2c414a700 2024-03-29T11:00:10.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.404+0000 7fc2ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:10.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.404+0000 7fc2ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:10.386 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:00:10.386 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 10 librados: watch_flush enter 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 10 librados: watch_flush exit 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 >> v1:172.21.15.107:6816/882194755 conn(0x7fc2a806dd30 legacy=0x7fc2a80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 10 monclient: shutdown 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 >> v1:172.21.15.107:6790/0 conn(0x7fc2c414a700 legacy=0x7fc2c41ee990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 shutdown_connections 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 >> 172.21.15.107:0/443162650 conn(0x7fc2c406b660 msgr2=0x7fc2c406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:10.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 shutdown_connections 2024-03-29T11:00:10.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.410+0000 7fc2cba74640 1 -- 172.21.15.107:0/443162650 wait complete. 2024-03-29T11:00:10.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:10.411+0000 7fc2cba74640 1 librados: shutdown 2024-03-29T11:00:10.407 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:17.214 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:17.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.390+0000 7fbe9402e640 10 monclient: get_monmap_and_config 2024-03-29T11:00:17.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.390+0000 7fbe9402e640 10 monclient: build_initial_monmap 2024-03-29T11:00:17.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.390+0000 7fbe9402e640 10 monclient: monmap: 2024-03-29T11:00:17.371 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:17.371 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:17.371 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:00:17.391399+0000 2024-03-29T11:00:17.372 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:00:17.391399+0000 2024-03-29T11:00:17.372 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:17.372 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:17.372 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:17.372 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:00:17.372 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:00:17.372 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:17.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.391+0000 7fbe9402e640 1 Processor -- start 2024-03-29T11:00:17.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.394+0000 7fbe9402e640 1 -- start start 2024-03-29T11:00:17.374 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.394+0000 7fbe9402e640 10 monclient: init 2024-03-29T11:00:17.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:17.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:17.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 10 monclient(hunting): picked mon.noname-c con 0x7fbe8c0a9ac0 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:17.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 10 monclient(hunting): picked mon.noname-b con 0x7fbe8c13ee90 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:17.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 10 monclient(hunting): picked mon.noname-a con 0x7fbe8c0af900 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:17.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8c0afd10 con 0x7fbe8c0af900 2024-03-29T11:00:17.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8c0af6f0 con 0x7fbe8c0a9ac0 2024-03-29T11:00:17.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8c0aac80 con 0x7fbe8c13ee90 2024-03-29T11:00:17.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:17.376 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.395+0000 7fbe9402e640 10 monclient(hunting): authenticate will time out at 948.663463s 2024-03-29T11:00:17.378 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.399+0000 7fbe925cb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbe8c0af900 0x7fbe8c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41161/0 (socket says 172.21.15.107:41161) 2024-03-29T11:00:17.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.399+0000 7fbe925cb640 1 -- 172.21.15.107:0/3569929393 learned_addr learned my addr 172.21.15.107:0/3569929393 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:17.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.399+0000 7fbe90dc8640 1 -- 172.21.15.107:0/3569929393 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007312287 0 0) 0x7fbe8c0afd10 con 0x7fbe8c0af900 2024-03-29T11:00:17.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.399+0000 7fbe90dc8640 10 monclient(hunting): _negotiate 2024-03-29T11:00:17.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.399+0000 7fbe90dc8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:17.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.399+0000 7fbe90dc8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:17.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.399+0000 7fbe90dc8640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:00:17.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.399+0000 7fbe90dc8640 1 -- 172.21.15.107:0/3569929393 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe78003520 con 0x7fbe8c0af900 2024-03-29T11:00:17.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.401+0000 7fbe90dc8640 1 -- 172.21.15.107:0/3569929393 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3034942797 0 0) 0x7fbe78003520 con 0x7fbe8c0af900 2024-03-29T11:00:17.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.401+0000 7fbe90dc8640 1 -- 172.21.15.107:0/3569929393 >> v1:172.21.15.107:6790/0 conn(0x7fbe8c13ee90 legacy=0x7fbe8c13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:17.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 1 -- 172.21.15.107:0/3569929393 >> v1:172.21.15.129:6789/0 conn(0x7fbe8c0a9ac0 legacy=0x7fbe8c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 10 monclient: _finish_hunting 0 2024-03-29T11:00:17.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 1 monclient: found mon.noname-a 2024-03-29T11:00:17.384 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 1 -- 172.21.15.107:0/3569929393 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c0aaf60 con 0x7fbe8c0af900 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 10 monclient: _finish_auth 0 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 10 monclient: _check_auth_tickets 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 1 -- 172.21.15.107:0/3569929393 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbe84001a10 con 0x7fbe8c0af900 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 10 monclient: dump: 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.385 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 1 -- 172.21.15.107:0/3569929393 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe84001be0 con 0x7fbe8c0af900 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.402+0000 7fbe90dc8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.403+0000 7fbe90dc8640 1 -- 172.21.15.107:0/3569929393 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbe84002030 con 0x7fbe8c0af900 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.403+0000 7fbe90dc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.403+0000 7fbe90dc8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.403+0000 7fbe90dc8640 10 monclient: dump: 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:17.386 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.403+0000 7fbe9402e640 5 monclient: authenticate success, global_id 4335 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.403+0000 7fbe9402e640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.403+0000 7fbe9402e640 4 set_mon_vals no callback set 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.404+0000 7fbe9402e640 10 monclient: shutdown 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.404+0000 7fbe9402e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.404+0000 7fbe9402e640 1 -- 172.21.15.107:0/3569929393 >> v1:172.21.15.107:6789/0 conn(0x7fbe8c0af900 legacy=0x7fbe8c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.404+0000 7fbe9402e640 1 -- 172.21.15.107:0/3569929393 shutdown_connections 2024-03-29T11:00:17.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.404+0000 7fbe9402e640 1 -- 172.21.15.107:0/3569929393 >> 172.21.15.107:0/3569929393 conn(0x7fbe8c0b35e0 msgr2=0x7fbe8c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:17.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.407+0000 7fbe9402e640 1 -- 172.21.15.107:0/3569929393 shutdown_connections 2024-03-29T11:00:17.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.407+0000 7fbe9402e640 1 -- 172.21.15.107:0/3569929393 wait complete. 2024-03-29T11:00:17.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.408+0000 7fbe9402e640 10 monclient: build_initial_monmap 2024-03-29T11:00:17.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.408+0000 7fbe9402e640 10 monclient: monmap: 2024-03-29T11:00:17.391 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:17.391 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:17.391 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.408+0000 7fbe9402e640 1 librados: starting msgr at 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.408+0000 7fbe9402e640 1 librados: starting objecter 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.408+0000 7fbe9402e640 1 Processor -- start 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.409+0000 7fbe9402e640 1 -- start start 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.409+0000 7fbe9402e640 1 librados: setting wanted keys 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.409+0000 7fbe9402e640 1 librados: calling monclient init 2024-03-29T11:00:17.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.409+0000 7fbe9402e640 10 monclient: init 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.409+0000 7fbe9402e640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.410+0000 7fbe9402e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.410+0000 7fbe9402e640 10 monclient(hunting): picked mon.noname-b con 0x7fbe8c0a9ac0 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.410+0000 7fbe9402e640 10 monclient(hunting): picked mon.noname-a con 0x7fbe8c0af900 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.410+0000 7fbe9402e640 10 monclient(hunting): picked mon.noname-c con 0x7fbe8c13ee90 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.410+0000 7fbe9402e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8c0703c0 con 0x7fbe8c0af900 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.410+0000 7fbe9402e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8c1d8a50 con 0x7fbe8c0a9ac0 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.410+0000 7fbe9402e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8c1d9640 con 0x7fbe8c13ee90 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.410+0000 7fbe9402e640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:17.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.410+0000 7fbe9402e640 10 monclient(hunting): authenticate will time out at 948.678077s 2024-03-29T11:00:17.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.413+0000 7fbe925cb640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fbe8c13ee90 0x7fbe8c071880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41633/0 (socket says 172.21.15.107:41633) 2024-03-29T11:00:17.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.413+0000 7fbe925cb640 1 -- 172.21.15.107:0/3554581891 learned_addr learned my addr 172.21.15.107:0/3554581891 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:17.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.414+0000 7fbe827fc640 10 client.?.objecter ms_handle_connect 0x7fbe8c13ee90 2024-03-29T11:00:17.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.414+0000 7fbe827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.415+0000 7fbe827fc640 10 client.?.objecter ms_handle_connect 0x7fbe8c0a9ac0 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.415+0000 7fbe827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.415+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290603707 0 0) 0x7fbe8c1d9640 con 0x7fbe8c13ee90 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.415+0000 7fbe827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.415+0000 7fbe827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.415+0000 7fbe827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.415+0000 7fbe827fc640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.415+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe70004eb0 con 0x7fbe8c13ee90 2024-03-29T11:00:17.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740333183 0 0) 0x7fbe8c1d8a50 con 0x7fbe8c0a9ac0 2024-03-29T11:00:17.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:17.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:17.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:17.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:00:17.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe8c1d9640 con 0x7fbe8c0a9ac0 2024-03-29T11:00:17.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3500066953 0 0) 0x7fbe70004eb0 con 0x7fbe8c13ee90 2024-03-29T11:00:17.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe8c1d8a50 con 0x7fbe8c13ee90 2024-03-29T11:00:17.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbe84001ef0 con 0x7fbe8c13ee90 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): dump: 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:17.405 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:17.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1048632966 0 0) 0x7fbe8c1d9640 con 0x7fbe8c0a9ac0 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe70004eb0 con 0x7fbe8c0a9ac0 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbe88001820 con 0x7fbe8c0a9ac0 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient(hunting): dump: 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.407 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4286743489 0 0) 0x7fbe8c1d8a50 con 0x7fbe8c13ee90 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 >> v1:172.21.15.129:6789/0 conn(0x7fbe8c0a9ac0 legacy=0x7fbe8c1d8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 >> v1:172.21.15.107:6789/0 conn(0x7fbe8c0af900 legacy=0x7fbe8c06e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:17.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 monclient: found mon.c 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c06eeb0 con 0x7fbe8c13ee90 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:17.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe9402e640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe9402e640 10 monclient: _renew_subs 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe9402e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8c06f100 con 0x7fbe8c13ee90 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe9402e640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe9402e640 10 monclient: _renew_subs 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe9402e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe8c1423f0 con 0x7fbe8c13ee90 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.416+0000 7fbe9402e640 1 librados: init done 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.419+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe84004f40 con 0x7fbe8c13ee90 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.419+0000 7fbe827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.419+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbe84005250 con 0x7fbe8c13ee90 2024-03-29T11:00:17.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.419+0000 7fbe827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.419+0000 7fbe827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.419+0000 7fbe827fc640 10 monclient: dump: 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.419+0000 7fbe837fe640 4 set_mon_vals no callback set 2024-03-29T11:00:17.411 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.420+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fbe8401b220 con 0x7fbe8c13ee90 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.420+0000 7fbe9402e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.420+0000 7fbe9402e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.421+0000 7fbe9402e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.421+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe74005080 con 0x7fbe8c13ee90 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.421+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fbe84084f90 con 0x7fbe8c13ee90 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.421+0000 7fbe827fc640 10 client.4340.objecter ms_dispatch 0x7fbe8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.421+0000 7fbe827fc640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.421+0000 7fbe827fc640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.421+0000 7fbe827fc640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.421+0000 7fbe91dca640 10 monclient: get_auth_request con 0x7fbe7007eea0 auth_method 0 2024-03-29T11:00:17.412 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.421+0000 7fbe827fc640 10 client.4340.objecter ms_handle_connect 0x7fbe7007eea0 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.432+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe840b28b0 con 0x7fbe8c13ee90 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.432+0000 7fbe827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:17.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.432+0000 7fbe827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:17.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.757+0000 7fbe9402e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:17.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.757+0000 7fbe9402e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:17.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.757+0000 7fbe9402e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:17.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.757+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe74005330 con 0x7fbe8c13ee90 2024-03-29T11:00:17.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.760+0000 7fbe827fc640 1 -- 172.21.15.107:0/3554581891 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fbe84050870 con 0x7fbe8c13ee90 2024-03-29T11:00:17.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.760+0000 7fbe827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:17.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.760+0000 7fbe827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:17.740 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:00:17.741 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:17.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.764+0000 7fbe9402e640 10 librados: watch_flush enter 2024-03-29T11:00:17.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.765+0000 7fbe9402e640 10 librados: watch_flush exit 2024-03-29T11:00:17.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.766+0000 7fbe9402e640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T11:00:17.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.766+0000 7fbe9402e640 10 client.4340.objecter successfully canceled tick 2024-03-29T11:00:17.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.766+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 >> v1:172.21.15.107:6816/882194755 conn(0x7fbe7007eea0 legacy=0x7fbe70081360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.766+0000 7fbe9402e640 10 monclient: shutdown 2024-03-29T11:00:17.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.766+0000 7fbe9402e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:17.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.766+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 >> v1:172.21.15.107:6790/0 conn(0x7fbe8c13ee90 legacy=0x7fbe8c071880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:17.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.766+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 shutdown_connections 2024-03-29T11:00:17.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.766+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 >> 172.21.15.107:0/3554581891 conn(0x7fbe8c0b35e0 msgr2=0x7fbe8c0ad060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:17.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.767+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 shutdown_connections 2024-03-29T11:00:17.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.767+0000 7fbe9402e640 1 -- 172.21.15.107:0/3554581891 wait complete. 2024-03-29T11:00:17.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:17.767+0000 7fbe9402e640 1 librados: shutdown 2024-03-29T11:00:17.767 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:24.573 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:24.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.756+0000 7f3a774ae640 10 monclient: get_monmap_and_config 2024-03-29T11:00:24.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.756+0000 7f3a774ae640 10 monclient: build_initial_monmap 2024-03-29T11:00:24.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.756+0000 7f3a774ae640 10 monclient: monmap: 2024-03-29T11:00:24.737 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:24.737 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:24.737 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:00:24.757415+0000 2024-03-29T11:00:24.737 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:00:24.757415+0000 2024-03-29T11:00:24.737 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:00:24.738 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:24.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.761+0000 7f3a774ae640 1 Processor -- start 2024-03-29T11:00:24.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.761+0000 7f3a774ae640 1 -- start start 2024-03-29T11:00:24.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.761+0000 7f3a774ae640 10 monclient: init 2024-03-29T11:00:24.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.762+0000 7f3a774ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:24.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.762+0000 7f3a774ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:24.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.762+0000 7f3a774ae640 10 monclient(hunting): picked mon.noname-c con 0x7f3a700b0860 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:24.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.762+0000 7f3a774ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3a7013ef30 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:24.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.764+0000 7f3a774ae640 10 monclient(hunting): picked mon.noname-b con 0x7f3a700af8e0 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:24.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.764+0000 7f3a774ae640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a700b0c40 con 0x7f3a7013ef30 2024-03-29T11:00:24.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.764+0000 7f3a774ae640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a700aed10 con 0x7f3a700b0860 2024-03-29T11:00:24.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.764+0000 7f3a774ae640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a700afd50 con 0x7f3a700af8e0 2024-03-29T11:00:24.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.764+0000 7f3a774ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:24.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.764+0000 7f3a774ae640 10 monclient(hunting): authenticate will time out at 956.032827s 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a75a4b640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3a700af8e0 0x7f3a70149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50973/0 (socket says 172.21.15.107:50973) 2024-03-29T11:00:24.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a75a4b640 1 -- 172.21.15.107:0/905121175 learned_addr learned my addr 172.21.15.107:0/905121175 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a67fff640 1 -- 172.21.15.107:0/905121175 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839238613 0 0) 0x7f3a700afd50 con 0x7f3a700af8e0 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a67fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a67fff640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a67fff640 1 -- 172.21.15.107:0/905121175 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a5c003520 con 0x7f3a700af8e0 2024-03-29T11:00:24.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a67fff640 1 -- 172.21.15.107:0/905121175 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1887468195 0 0) 0x7f3a5c003520 con 0x7f3a700af8e0 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.766+0000 7f3a67fff640 1 -- 172.21.15.107:0/905121175 >> v1:172.21.15.129:6789/0 conn(0x7f3a700b0860 legacy=0x7f3a7013e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:24.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.767+0000 7f3a67fff640 1 -- 172.21.15.107:0/905121175 >> v1:172.21.15.107:6789/0 conn(0x7f3a7013ef30 legacy=0x7f3a700af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 1 monclient: found mon.noname-b 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 1 -- 172.21.15.107:0/905121175 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a700ab1b0 con 0x7f3a700af8e0 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 1 -- 172.21.15.107:0/905121175 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3a68001a10 con 0x7f3a700af8e0 2024-03-29T11:00:24.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a67fff640 10 monclient: dump: 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:24.751 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a774ae640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:00:24.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.769+0000 7f3a774ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.774+0000 7f3a67fff640 1 -- 172.21.15.107:0/905121175 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a68001be0 con 0x7f3a700af8e0 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.774+0000 7f3a67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.774+0000 7f3a67fff640 1 -- 172.21.15.107:0/905121175 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3a68002030 con 0x7f3a700af8e0 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.774+0000 7f3a774ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:24.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.774+0000 7f3a774ae640 4 set_mon_vals no callback set 2024-03-29T11:00:24.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.775+0000 7f3a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.775+0000 7f3a67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:24.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.775+0000 7f3a67fff640 10 monclient: dump: 2024-03-29T11:00:24.758 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:24.758 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:24.758 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.758 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.775+0000 7f3a774ae640 10 monclient: shutdown 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.775+0000 7f3a774ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.775+0000 7f3a774ae640 1 -- 172.21.15.107:0/905121175 >> v1:172.21.15.107:6790/0 conn(0x7f3a700af8e0 legacy=0x7f3a70149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.775+0000 7f3a774ae640 1 -- 172.21.15.107:0/905121175 shutdown_connections 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.775+0000 7f3a774ae640 1 -- 172.21.15.107:0/905121175 >> 172.21.15.107:0/905121175 conn(0x7f3a7006b660 msgr2=0x7f3a7006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:24.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.776+0000 7f3a774ae640 1 -- 172.21.15.107:0/905121175 shutdown_connections 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.779+0000 7f3a774ae640 1 -- 172.21.15.107:0/905121175 wait complete. 2024-03-29T11:00:24.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.780+0000 7f3a774ae640 10 monclient: build_initial_monmap 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.781+0000 7f3a774ae640 10 monclient: monmap: 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:24.761 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:24.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.781+0000 7f3a774ae640 1 librados: starting msgr at 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.781+0000 7f3a774ae640 1 librados: starting objecter 2024-03-29T11:00:24.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.781+0000 7f3a774ae640 1 Processor -- start 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.782+0000 7f3a774ae640 1 -- start start 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.782+0000 7f3a774ae640 1 librados: setting wanted keys 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.782+0000 7f3a774ae640 1 librados: calling monclient init 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.782+0000 7f3a774ae640 10 monclient: init 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3a700af8e0 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 10 monclient(hunting): picked mon.noname-b con 0x7f3a700b0860 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 10 monclient(hunting): picked mon.noname-c con 0x7f3a7013ef30 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7014a050 con 0x7f3a700af8e0 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a70148d20 con 0x7f3a700b0860 2024-03-29T11:00:24.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7014a4e0 con 0x7f3a7013ef30 2024-03-29T11:00:24.766 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:24.766 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.783+0000 7f3a774ae640 10 monclient(hunting): authenticate will time out at 956.051165s 2024-03-29T11:00:24.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.789+0000 7f3a75a4b640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3a7013ef30 0x7f3a701e6650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51121/0 (socket says 172.21.15.107:51121) 2024-03-29T11:00:24.769 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.789+0000 7f3a75a4b640 1 -- 172.21.15.107:0/1181570006 learned_addr learned my addr 172.21.15.107:0/1181570006 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:24.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.790+0000 7f3a65ffb640 10 client.?.objecter ms_handle_connect 0x7f3a700af8e0 2024-03-29T11:00:24.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.790+0000 7f3a65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:24.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.790+0000 7f3a65ffb640 10 client.?.objecter ms_handle_connect 0x7f3a7013ef30 2024-03-29T11:00:24.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.790+0000 7f3a65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:24.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.790+0000 7f3a65ffb640 10 client.?.objecter ms_handle_connect 0x7f3a700b0860 2024-03-29T11:00:24.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.790+0000 7f3a65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.790+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727377557 0 0) 0x7f3a70148d20 con 0x7f3a700b0860 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a54004eb0 con 0x7f3a700b0860 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136165871 0 0) 0x7f3a7014a4e0 con 0x7f3a7013ef30 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.778 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:00:24.778 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a70148d20 con 0x7f3a7013ef30 2024-03-29T11:00:24.778 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749059285 0 0) 0x7f3a7014a050 con 0x7f3a700af8e0 2024-03-29T11:00:24.778 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:24.778 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:24.778 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:24.778 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:00:24.778 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a7014a4e0 con 0x7f3a700af8e0 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1688811224 0 0) 0x7f3a54004eb0 con 0x7f3a700b0860 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a7014a050 con 0x7f3a700b0860 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3a600017d0 con 0x7f3a700b0860 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.791+0000 7f3a65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.779 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:24.780 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:24.780 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:24.780 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:24.780 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:24.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:24.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3388443366 0 0) 0x7f3a70148d20 con 0x7f3a7013ef30 2024-03-29T11:00:24.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a54004eb0 con 0x7f3a7013ef30 2024-03-29T11:00:24.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3a68001ef0 con 0x7f3a7013ef30 2024-03-29T11:00:24.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:24.781 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:24.781 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:24.781 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:24.782 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.782 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.782 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.782 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:24.782 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:24.782 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:24.782 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:24.782 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3990352415 0 0) 0x7f3a7014a050 con 0x7f3a700b0860 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 >> v1:172.21.15.107:6790/0 conn(0x7f3a7013ef30 legacy=0x7f3a701e6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 >> v1:172.21.15.107:6789/0 conn(0x7f3a700af8e0 legacy=0x7f3a701494b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 1 monclient: found mon.b 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.792+0000 7f3a65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a65ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7014a670 con 0x7f3a700b0860 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a65ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a65ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:24.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a774ae640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a774ae640 10 monclient: _renew_subs 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a774ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a70149bc0 con 0x7f3a700b0860 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a774ae640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a774ae640 10 monclient: _renew_subs 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a774ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7014b370 con 0x7f3a700b0860 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.793+0000 7f3a774ae640 1 librados: init done 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.795+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a60001930 con 0x7f3a700b0860 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.795+0000 7f3a65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.795+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3a60003790 con 0x7f3a700b0860 2024-03-29T11:00:24.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.795+0000 7f3a65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.795+0000 7f3a65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.795+0000 7f3a65ffb640 10 monclient: dump: 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:24.785 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:24.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.797+0000 7f3a774ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:24.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.797+0000 7f3a774ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:24.787 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.797+0000 7f3a774ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.797+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a70148b60 con 0x7f3a700b0860 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.797+0000 7f3a677fe640 4 set_mon_vals no callback set 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.798+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f3a60019760 con 0x7f3a700b0860 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.800+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f3a600836b0 con 0x7f3a700b0860 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.800+0000 7f3a65ffb640 10 client.4333.objecter ms_dispatch 0x7f3a70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.800+0000 7f3a65ffb640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.800+0000 7f3a65ffb640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.800+0000 7f3a65ffb640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.803+0000 7f3a7524a640 10 monclient: get_auth_request con 0x7f3a5406e070 auth_method 0 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.804+0000 7f3a65ffb640 10 client.4333.objecter ms_handle_connect 0x7f3a5406e070 2024-03-29T11:00:24.788 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.807+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a6004ef90 con 0x7f3a700b0860 2024-03-29T11:00:24.789 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.807+0000 7f3a65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:24.789 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:24.807+0000 7f3a65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:25.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.127+0000 7f3a774ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:25.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.127+0000 7f3a774ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:25.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.127+0000 7f3a774ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:00:25.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.127+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a7014a200 con 0x7f3a700b0860 2024-03-29T11:00:25.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.128+0000 7f3a65ffb640 1 -- 172.21.15.107:0/1181570006 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f3a7014a200 con 0x7f3a700b0860 2024-03-29T11:00:25.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.128+0000 7f3a65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:25.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.128+0000 7f3a65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:25.109 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:00:25.109 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:25.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.132+0000 7f3a774ae640 10 librados: watch_flush enter 2024-03-29T11:00:25.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.132+0000 7f3a774ae640 10 librados: watch_flush exit 2024-03-29T11:00:25.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.132+0000 7f3a774ae640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T11:00:25.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.132+0000 7f3a774ae640 10 client.4333.objecter successfully canceled tick 2024-03-29T11:00:25.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.134+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 >> v1:172.21.15.107:6816/882194755 conn(0x7f3a5406e070 legacy=0x7f3a54070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:25.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.134+0000 7f3a774ae640 10 monclient: shutdown 2024-03-29T11:00:25.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.134+0000 7f3a774ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:25.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.134+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 >> v1:172.21.15.129:6789/0 conn(0x7f3a700b0860 legacy=0x7f3a70136bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:25.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.136+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 shutdown_connections 2024-03-29T11:00:25.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.136+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 >> 172.21.15.107:0/1181570006 conn(0x7f3a7006b660 msgr2=0x7f3a7013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:25.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.137+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 shutdown_connections 2024-03-29T11:00:25.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.140+0000 7f3a774ae640 1 -- 172.21.15.107:0/1181570006 wait complete. 2024-03-29T11:00:25.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:25.140+0000 7f3a774ae640 1 librados: shutdown 2024-03-29T11:00:25.137 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:31.944 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.119+0000 7f79b9344640 10 monclient: get_monmap_and_config 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.119+0000 7f79b9344640 10 monclient: build_initial_monmap 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.119+0000 7f79b9344640 10 monclient: monmap: 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:00:32.120218+0000 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:00:32.120218+0000 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:32.099 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:32.100 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:00:32.100 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:00:32.100 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:32.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.120+0000 7f79b9344640 1 Processor -- start 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.121+0000 7f79b9344640 1 -- start start 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.121+0000 7f79b9344640 10 monclient: init 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.121+0000 7f79b9344640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.121+0000 7f79b9344640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.121+0000 7f79b9344640 10 monclient(hunting): picked mon.noname-c con 0x7f79b40b1f60 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.121+0000 7f79b9344640 10 monclient(hunting): picked mon.noname-a con 0x7f79b4071eb0 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.121+0000 7f79b9344640 10 monclient(hunting): picked mon.noname-b con 0x7f79b4143d10 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.121+0000 7f79b9344640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b406ef80 con 0x7f79b4071eb0 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.122+0000 7f79b9344640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b406f260 con 0x7f79b40b1f60 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.122+0000 7f79b9344640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b40b40c0 con 0x7f79b4143d10 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.122+0000 7f79b9344640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:32.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.122+0000 7f79b9344640 10 monclient(hunting): authenticate will time out at 963.390005s 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.122+0000 7f79b27fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f79b4071eb0 0x7f79b4146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59692/0 (socket says 172.21.15.107:59692) 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.122+0000 7f79b27fc640 1 -- 172.21.15.107:0/3569921948 learned_addr learned my addr 172.21.15.107:0/3569921948 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.123+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992081144 0 0) 0x7f79b406ef80 con 0x7f79b4071eb0 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.123+0000 7f79b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.123+0000 7f79b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.123+0000 7f79b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.123+0000 7f79b1ffb640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.123+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f799c003520 con 0x7f79b4071eb0 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139653599 0 0) 0x7f79b40b40c0 con 0x7f79b4143d10 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:00:32.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79b406ef80 con 0x7f79b4143d10 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1558203672 0 0) 0x7f799c003520 con 0x7f79b4071eb0 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 >> v1:172.21.15.107:6790/0 conn(0x7f79b4143d10 legacy=0x7f79b4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 >> v1:172.21.15.129:6789/0 conn(0x7f79b40b1f60 legacy=0x7f79b4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:32.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b4155d70 con 0x7f79b4071eb0 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f79ac001a10 con 0x7f79b4071eb0 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b9344640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b9344640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b1ffb640 10 monclient: dump: 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:32.106 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.124+0000 7f79b9344640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.125+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79ac001b70 con 0x7f79b4071eb0 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.125+0000 7f79b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.125+0000 7f79b9344640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.125+0000 7f79b9344640 4 set_mon_vals no callback set 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.125+0000 7f79b1ffb640 1 -- 172.21.15.107:0/3569921948 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f79ac001f80 con 0x7f79b4071eb0 2024-03-29T11:00:32.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.125+0000 7f79b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.125+0000 7f79b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.125+0000 7f79b1ffb640 10 monclient: dump: 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:32.108 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.126+0000 7f79b9344640 10 monclient: shutdown 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.126+0000 7f79b9344640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.126+0000 7f79b9344640 1 -- 172.21.15.107:0/3569921948 >> v1:172.21.15.107:6789/0 conn(0x7f79b4071eb0 legacy=0x7f79b4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.128+0000 7f79b9344640 1 -- 172.21.15.107:0/3569921948 shutdown_connections 2024-03-29T11:00:32.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.128+0000 7f79b9344640 1 -- 172.21.15.107:0/3569921948 >> 172.21.15.107:0/3569921948 conn(0x7f79b4066a00 msgr2=0x7f79b4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.129+0000 7f79b9344640 1 -- 172.21.15.107:0/3569921948 shutdown_connections 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.129+0000 7f79b9344640 1 -- 172.21.15.107:0/3569921948 wait complete. 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.130+0000 7f79b9344640 10 monclient: build_initial_monmap 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.130+0000 7f79b9344640 10 monclient: monmap: 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:32.110 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:32.111 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:00:32.111 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:00:32.111 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:32.111 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:32.111 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:32.111 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:00:32.111 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:00:32.111 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:32.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.131+0000 7f79b9344640 1 librados: starting msgr at 2024-03-29T11:00:32.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.131+0000 7f79b9344640 1 librados: starting objecter 2024-03-29T11:00:32.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.131+0000 7f79b9344640 1 Processor -- start 2024-03-29T11:00:32.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.131+0000 7f79b9344640 1 -- start start 2024-03-29T11:00:32.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.131+0000 7f79b9344640 1 librados: setting wanted keys 2024-03-29T11:00:32.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.131+0000 7f79b9344640 1 librados: calling monclient init 2024-03-29T11:00:32.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.131+0000 7f79b9344640 10 monclient: init 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 10 monclient(hunting): picked mon.noname-a con 0x7f79b40b1f60 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 10 monclient(hunting): picked mon.noname-c con 0x7f79b4143d10 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 10 monclient(hunting): picked mon.noname-b con 0x7f79b4147ca0 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b41487c0 con 0x7f79b40b1f60 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b413b940 con 0x7f79b4147ca0 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b414b4f0 con 0x7f79b4143d10 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b9344640 10 monclient(hunting): authenticate will time out at 963.400430s 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f79b40b1f60 0x7f79b4148e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60270/0 (socket says 172.21.15.107:60270) 2024-03-29T11:00:32.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.132+0000 7f79b2ffd640 1 -- 172.21.15.107:0/3618119280 learned_addr learned my addr 172.21.15.107:0/3618119280 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.133+0000 7f799b7fe640 10 client.?.objecter ms_handle_connect 0x7f79b40b1f60 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.133+0000 7f799b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.133+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435546005 0 0) 0x7f79a80011a0 con 0x7f79b40b1f60 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.133+0000 7f799b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.133+0000 7f799b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.133+0000 7f799b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.133+0000 7f799b7fe640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.133+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a0004eb0 con 0x7f79b40b1f60 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2213720858 0 0) 0x7f79a0004eb0 con 0x7f79b40b1f60 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79a80011a0 con 0x7f79b40b1f60 2024-03-29T11:00:32.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f79a8001950 con 0x7f79b40b1f60 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 10 monclient(hunting): dump: 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:32.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3319850007 0 0) 0x7f79a80011a0 con 0x7f79b40b1f60 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 >> v1:172.21.15.107:6790/0 conn(0x7f79b4143d10 legacy=0x7f79b413ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.134+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 >> v1:172.21.15.129:6789/0 conn(0x7f79b4147ca0 legacy=0x7f79b413b230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 1 monclient: found mon.a 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b41482a0 con 0x7f79b40b1f60 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 10 client.?.objecter ms_handle_connect 0x7f79b4143d10 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f799b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f79b9344640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:00:32.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f79b9344640 10 monclient: _renew_subs 2024-03-29T11:00:32.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.135+0000 7f79b9344640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:32.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.136+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b413f200 con 0x7f79b40b1f60 2024-03-29T11:00:32.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.136+0000 7f79b9344640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:32.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.136+0000 7f79b9344640 10 monclient: _renew_subs 2024-03-29T11:00:32.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.136+0000 7f79b9344640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:32.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.136+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b4070d40 con 0x7f79b40b1f60 2024-03-29T11:00:32.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.136+0000 7f79b9344640 1 librados: init done 2024-03-29T11:00:32.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.137+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79a8001c70 con 0x7f79b40b1f60 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.137+0000 7f799b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.137+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f79a8003a40 con 0x7f79b40b1f60 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.137+0000 7f799b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.137+0000 7f799b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.137+0000 7f799b7fe640 10 monclient: dump: 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:32.123 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.137+0000 7f79b0ff9640 4 set_mon_vals no callback set 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.138+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f79a8003c60 con 0x7f79b40b1f60 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.138+0000 7f79b9344640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.138+0000 7f79b9344640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.138+0000 7f79b9344640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.138+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7994005080 con 0x7f79b40b1f60 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.139+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f79a8084a90 con 0x7f79b40b1f60 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.139+0000 7f799b7fe640 10 client.4344.objecter ms_dispatch 0x7f79b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.139+0000 7f799b7fe640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.139+0000 7f799b7fe640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.139+0000 7f79b27fc640 10 monclient: get_auth_request con 0x7f79a006dc20 auth_method 0 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.139+0000 7f799b7fe640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T11:00:32.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.139+0000 7f799b7fe640 10 client.4344.objecter ms_handle_connect 0x7f79a006dc20 2024-03-29T11:00:32.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.153+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79a80503d0 con 0x7f79b40b1f60 2024-03-29T11:00:32.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.153+0000 7f799b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.153+0000 7f799b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:32.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.503+0000 7f79b9344640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:32.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.503+0000 7f79b9344640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:32.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.503+0000 7f79b9344640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:32.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.503+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7994005330 con 0x7f79b40b1f60 2024-03-29T11:00:32.484 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:00:32.484 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:32.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.504+0000 7f799b7fe640 1 -- 172.21.15.107:0/3618119280 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f79a8084140 con 0x7f79b40b1f60 2024-03-29T11:00:32.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.504+0000 7f799b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:32.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.504+0000 7f799b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:32.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 10 librados: watch_flush enter 2024-03-29T11:00:32.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 10 librados: watch_flush exit 2024-03-29T11:00:32.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T11:00:32.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 10 client.4344.objecter successfully canceled tick 2024-03-29T11:00:32.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 >> v1:172.21.15.107:6816/882194755 conn(0x7f79a006dc20 legacy=0x7f79a00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 10 monclient: shutdown 2024-03-29T11:00:32.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:32.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 >> v1:172.21.15.107:6789/0 conn(0x7f79b40b1f60 legacy=0x7f79b4148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 shutdown_connections 2024-03-29T11:00:32.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.508+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 >> 172.21.15.107:0/3618119280 conn(0x7f79b4066a00 msgr2=0x7f79b4142900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:32.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.511+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 shutdown_connections 2024-03-29T11:00:32.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.511+0000 7f79b9344640 1 -- 172.21.15.107:0/3618119280 wait complete. 2024-03-29T11:00:32.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:32.511+0000 7f79b9344640 1 librados: shutdown 2024-03-29T11:00:32.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:39.314 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:39.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.474+0000 7fc019e86640 10 monclient: get_monmap_and_config 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.474+0000 7fc019e86640 10 monclient: build_initial_monmap 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.474+0000 7fc019e86640 10 monclient: monmap: 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:00:39.475586+0000 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:00:39.475586+0000 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:00:39.455 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:39.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.475+0000 7fc019e86640 1 Processor -- start 2024-03-29T11:00:39.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.475+0000 7fc019e86640 1 -- start start 2024-03-29T11:00:39.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.475+0000 7fc019e86640 10 monclient: init 2024-03-29T11:00:39.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.476+0000 7fc019e86640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:39.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.476+0000 7fc019e86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:39.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.476+0000 7fc019e86640 10 monclient(hunting): picked mon.noname-c con 0x7fc0141422b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:39.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.476+0000 7fc019e86640 10 monclient(hunting): picked mon.noname-a con 0x7fc014142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:39.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.476+0000 7fc019e86640 10 monclient(hunting): picked mon.noname-b con 0x7fc0140b17d0 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:39.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.476+0000 7fc019e86640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc014142c80 con 0x7fc014142690 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.477+0000 7fc012ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc014142690 0x7fc0140b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47768/0 (socket says 172.21.15.107:47768) 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.477+0000 7fc012ffd640 1 -- 172.21.15.107:0/276531843 learned_addr learned my addr 172.21.15.107:0/276531843 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.476+0000 7fc019e86640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0140aae40 con 0x7fc0141422b0 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.477+0000 7fc019e86640 1 -- 172.21.15.107:0/276531843 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01413fbd0 con 0x7fc0140b17d0 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.477+0000 7fc019e86640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.477+0000 7fc019e86640 10 monclient(hunting): authenticate will time out at 970.745419s 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22910412 0 0) 0x7fc014142c80 con 0x7fc014142690 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc000003520 con 0x7fc014142690 2024-03-29T11:00:39.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027426657 0 0) 0x7fc0140aae40 con 0x7fc0141422b0 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc014142c80 con 0x7fc0141422b0 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 630451343 0 0) 0x7fc014142c80 con 0x7fc0141422b0 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 >> v1:172.21.15.107:6790/0 conn(0x7fc0140b17d0 legacy=0x7fc0140b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 >> v1:172.21.15.107:6789/0 conn(0x7fc014142690 legacy=0x7fc0140b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 monclient: found mon.noname-c 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014150930 con 0x7fc0141422b0 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:39.461 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc019e86640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc019e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc00c001630 con 0x7fc0141422b0 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc0127fc640 10 monclient: dump: 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.462 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.478+0000 7fc019e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.479+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc00c001810 con 0x7fc0141422b0 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.479+0000 7fc0127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.479+0000 7fc019e86640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.479+0000 7fc019e86640 4 set_mon_vals no callback set 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.479+0000 7fc0127fc640 1 -- 172.21.15.107:0/276531843 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc00c001630 con 0x7fc0141422b0 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.479+0000 7fc0127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.479+0000 7fc0127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.479+0000 7fc0127fc640 10 monclient: dump: 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:39.463 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:39.464 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.464 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.464 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.464 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:39.464 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:39.464 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:39.464 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:39.464 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:39.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.480+0000 7fc019e86640 10 monclient: shutdown 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.480+0000 7fc019e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.480+0000 7fc019e86640 1 -- 172.21.15.107:0/276531843 >> v1:172.21.15.129:6789/0 conn(0x7fc0141422b0 legacy=0x7fc01413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.480+0000 7fc019e86640 1 -- 172.21.15.107:0/276531843 shutdown_connections 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.480+0000 7fc019e86640 1 -- 172.21.15.107:0/276531843 >> 172.21.15.107:0/276531843 conn(0x7fc014066a00 msgr2=0x7fc014066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.480+0000 7fc019e86640 1 -- 172.21.15.107:0/276531843 shutdown_connections 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.480+0000 7fc019e86640 1 -- 172.21.15.107:0/276531843 wait complete. 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.481+0000 7fc019e86640 10 monclient: build_initial_monmap 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.481+0000 7fc019e86640 10 monclient: monmap: 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:00:39.468 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.481+0000 7fc019e86640 1 librados: starting msgr at 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.481+0000 7fc019e86640 1 librados: starting objecter 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.482+0000 7fc019e86640 1 Processor -- start 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.482+0000 7fc019e86640 1 -- start start 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.482+0000 7fc019e86640 1 librados: setting wanted keys 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.482+0000 7fc019e86640 1 librados: calling monclient init 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.482+0000 7fc019e86640 10 monclient: init 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:39.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 10 monclient(hunting): picked mon.noname-c con 0x7fc0140b17d0 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 10 monclient(hunting): picked mon.noname-a con 0x7fc014142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 10 monclient(hunting): picked mon.noname-b con 0x7fc014132230 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0140b5780 con 0x7fc014142690 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc014133000 con 0x7fc014132230 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0141332e0 con 0x7fc0140b17d0 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc019e86640 10 monclient(hunting): authenticate will time out at 970.751249s 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc012ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc014142690 0x7fc01412b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47946/0 (socket says 172.21.15.107:47946) 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc012ffd640 1 -- 172.21.15.107:0/3702118649 learned_addr learned my addr 172.21.15.107:0/3702118649 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 10 client.?.objecter ms_handle_connect 0x7fc014142690 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953344793 0 0) 0x7fc0140b5780 con 0x7fc014142690 2024-03-29T11:00:39.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:00:39.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff4004eb0 con 0x7fc014142690 2024-03-29T11:00:39.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 10 client.?.objecter ms_handle_connect 0x7fc0140b17d0 2024-03-29T11:00:39.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.483+0000 7fc00bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:39.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398624443 0 0) 0x7fc0141332e0 con 0x7fc0140b17d0 2024-03-29T11:00:39.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:39.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0140b5780 con 0x7fc0140b17d0 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1936883111 0 0) 0x7fbff4004eb0 con 0x7fc014142690 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0141332e0 con 0x7fc014142690 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc004001bb0 con 0x7fc014142690 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): dump: 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:39.472 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 client.?.objecter ms_handle_connect 0x7fc014132230 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2460148817 0 0) 0x7fc0140b5780 con 0x7fc0140b17d0 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbff4004eb0 con 0x7fc0140b17d0 2024-03-29T11:00:39.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc00c0015f0 con 0x7fc0140b17d0 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.484+0000 7fc00bfff640 10 monclient(hunting): dump: 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:39.474 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2342037415 0 0) 0x7fc0141332e0 con 0x7fc014142690 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 >> v1:172.21.15.107:6790/0 conn(0x7fc0140b17d0 legacy=0x7fc0141317d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 >> v1:172.21.15.129:6789/0 conn(0x7fc014132230 legacy=0x7fc014071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 1 monclient: found mon.a 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01412bc90 con 0x7fc014142690 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 10 monclient: _finish_auth 0 2024-03-29T11:00:39.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc00bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc019e86640 5 monclient: authenticate success, global_id 4350 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc019e86640 10 monclient: _renew_subs 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc019e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01412bee0 con 0x7fc014142690 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc019e86640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc019e86640 10 monclient: _renew_subs 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc019e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc014071b20 con 0x7fc014142690 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.485+0000 7fc019e86640 1 librados: init done 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.486+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc004004b60 con 0x7fc014142690 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.486+0000 7fc00bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:39.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.486+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc004004e70 con 0x7fc014142690 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.486+0000 7fc00bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.486+0000 7fc00bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.486+0000 7fc011ffb640 4 set_mon_vals no callback set 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.486+0000 7fc00bfff640 10 monclient: dump: 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:39.477 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:39.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.487+0000 7fc019e86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:39.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.487+0000 7fc019e86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:39.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.487+0000 7fc019e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:39.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.487+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff8005080 con 0x7fc014142690 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.489+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fc00401ae40 con 0x7fc014142690 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.489+0000 7fc0137fe640 10 monclient: get_auth_request con 0x7fbff406dd90 auth_method 0 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.489+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fc004084ce0 con 0x7fc014142690 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.489+0000 7fc00bfff640 10 client.4350.objecter ms_dispatch 0x7fc014000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.489+0000 7fc00bfff640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.489+0000 7fc00bfff640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.490+0000 7fc00bfff640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.490+0000 7fc00bfff640 10 client.4350.objecter ms_handle_connect 0x7fbff406dd90 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.498+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0040505c0 con 0x7fc014142690 2024-03-29T11:00:39.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.499+0000 7fc00bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:39.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.499+0000 7fc00bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:39.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.853+0000 7fc019e86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:39.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.853+0000 7fc019e86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:39.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.853+0000 7fc019e86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:00:39.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.853+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbff8005330 con 0x7fc014142690 2024-03-29T11:00:39.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.854+0000 7fc00bfff640 1 -- 172.21.15.107:0/3702118649 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fc0040524f0 con 0x7fc014142690 2024-03-29T11:00:39.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.854+0000 7fc00bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:39.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.854+0000 7fc00bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:39.835 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:00:39.835 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:39.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.857+0000 7fc019e86640 10 librados: watch_flush enter 2024-03-29T11:00:39.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 10 librados: watch_flush exit 2024-03-29T11:00:39.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-29T11:00:39.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 10 client.4350.objecter successfully canceled tick 2024-03-29T11:00:39.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 >> v1:172.21.15.107:6816/882194755 conn(0x7fbff406dd90 legacy=0x7fbff4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 10 monclient: shutdown 2024-03-29T11:00:39.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:39.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 >> v1:172.21.15.107:6789/0 conn(0x7fc014142690 legacy=0x7fc01412b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:39.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 shutdown_connections 2024-03-29T11:00:39.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 >> 172.21.15.107:0/3702118649 conn(0x7fc014066a00 msgr2=0x7fc01413f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:39.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 shutdown_connections 2024-03-29T11:00:39.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 1 -- 172.21.15.107:0/3702118649 wait complete. 2024-03-29T11:00:39.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:39.858+0000 7fc019e86640 1 librados: shutdown 2024-03-29T11:00:39.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:46.660 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:46.792 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.812+0000 7f28d8ee4640 10 monclient: get_monmap_and_config 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.812+0000 7f28d8ee4640 10 monclient: build_initial_monmap 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.812+0000 7f28d8ee4640 10 monclient: monmap: 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:00:46.813840+0000 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:00:46.813840+0000 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:00:46.793 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:00:46.794 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.813+0000 7f28d8ee4640 1 Processor -- start 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.813+0000 7f28d8ee4640 1 -- start start 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.813+0000 7f28d8ee4640 10 monclient: init 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f28d40b4330 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f28d41470b0 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f28d41439b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d406f2a0 con 0x7f28d41470b0 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d4146b90 con 0x7f28d40b4330 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d40b2ca0 con 0x7f28d41439b0 2024-03-29T11:00:46.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.814+0000 7f28d8ee4640 10 monclient(hunting): authenticate will time out at 978.082928s 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d2d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f28d41439b0 0x7f28d4151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36940/0 (socket says 172.21.15.107:36940) 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d2d9d640 1 -- 172.21.15.107:0/3937916950 learned_addr learned my addr 172.21.15.107:0/3937916950 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d159a640 1 -- 172.21.15.107:0/3937916950 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576796170 0 0) 0x7f28d40b2ca0 con 0x7f28d41439b0 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d159a640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d159a640 1 -- 172.21.15.107:0/3937916950 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c0003520 con 0x7f28d41439b0 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d159a640 1 -- 172.21.15.107:0/3937916950 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3830375169 0 0) 0x7f28c0003520 con 0x7f28d41439b0 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d159a640 1 -- 172.21.15.107:0/3937916950 >> v1:172.21.15.129:6789/0 conn(0x7f28d40b4330 legacy=0x7f28d40b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:46.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.815+0000 7f28d159a640 1 -- 172.21.15.107:0/3937916950 >> v1:172.21.15.107:6789/0 conn(0x7f28d41470b0 legacy=0x7f28d40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:46.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: _finish_hunting 0 2024-03-29T11:00:46.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 1 monclient: found mon.noname-b 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 1 -- 172.21.15.107:0/3937916950 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d406e0c0 con 0x7f28d41439b0 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: _finish_auth 0 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: _check_auth_tickets 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 1 -- 172.21.15.107:0/3937916950 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f28cc001690 con 0x7f28d41439b0 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: dump: 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:46.799 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d8ee4640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d8ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 1 -- 172.21.15.107:0/3937916950 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28cc0019a0 con 0x7f28d41439b0 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 1 -- 172.21.15.107:0/3937916950 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f28cc001d30 con 0x7f28d41439b0 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d159a640 10 monclient: dump: 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d8ee4640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:46.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.818+0000 7f28d8ee4640 4 set_mon_vals no callback set 2024-03-29T11:00:46.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.819+0000 7f28d8ee4640 10 monclient: shutdown 2024-03-29T11:00:46.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.819+0000 7f28d8ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:46.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.819+0000 7f28d8ee4640 1 -- 172.21.15.107:0/3937916950 >> v1:172.21.15.107:6790/0 conn(0x7f28d41439b0 legacy=0x7f28d4151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.819+0000 7f28d8ee4640 1 -- 172.21.15.107:0/3937916950 shutdown_connections 2024-03-29T11:00:46.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.819+0000 7f28d8ee4640 1 -- 172.21.15.107:0/3937916950 >> 172.21.15.107:0/3937916950 conn(0x7f28d4066a00 msgr2=0x7f28d4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:46.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.819+0000 7f28d8ee4640 1 -- 172.21.15.107:0/3937916950 shutdown_connections 2024-03-29T11:00:46.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.820+0000 7f28d8ee4640 1 -- 172.21.15.107:0/3937916950 wait complete. 2024-03-29T11:00:46.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.821+0000 7f28d8ee4640 10 monclient: build_initial_monmap 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.821+0000 7f28d8ee4640 10 monclient: monmap: 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.821+0000 7f28d8ee4640 1 librados: starting msgr at 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.821+0000 7f28d8ee4640 1 librados: starting objecter 2024-03-29T11:00:46.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.821+0000 7f28d8ee4640 1 Processor -- start 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.821+0000 7f28d8ee4640 1 -- start start 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.821+0000 7f28d8ee4640 1 librados: setting wanted keys 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.821+0000 7f28d8ee4640 1 librados: calling monclient init 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.821+0000 7f28d8ee4640 10 monclient: init 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.822+0000 7f28d8ee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.822+0000 7f28d8ee4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.822+0000 7f28d8ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f28d40b4330 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.822+0000 7f28d8ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f28d41439b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28d8ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f28d41470b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:46.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28d8ee4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d411f770 con 0x7f28d40b4330 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28d8ee4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d4121980 con 0x7f28d41439b0 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28d2d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f28d41470b0 0x7f28d41e6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37046/0 (socket says 172.21.15.107:37046) 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28d2d9d640 1 -- 172.21.15.107:0/1106419199 learned_addr learned my addr 172.21.15.107:0/1106419199 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d41d8eb0 con 0x7f28d41470b0 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28d8ee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28d8ee4640 10 monclient(hunting): authenticate will time out at 978.091354s 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28caffd640 10 client.?.objecter ms_handle_connect 0x7f28d41470b0 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303523131 0 0) 0x7f28d41d8eb0 con 0x7f28d41470b0 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:46.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28caffd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.823+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b4004eb0 con 0x7f28d41470b0 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3687207092 0 0) 0x7f28b4004eb0 con 0x7f28d41470b0 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28d41d8eb0 con 0x7f28d41470b0 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f28cc001650 con 0x7f28d41470b0 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 10 monclient(hunting): dump: 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:46.812 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2229088472 0 0) 0x7f28d41d8eb0 con 0x7f28d41470b0 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 >> v1:172.21.15.129:6789/0 conn(0x7f28d41439b0 legacy=0x7f28d4140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.824+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 >> v1:172.21.15.107:6789/0 conn(0x7f28d40b4330 legacy=0x7f28d411ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.825+0000 7f28caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.825+0000 7f28caffd640 1 monclient: found mon.c 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.825+0000 7f28caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.825+0000 7f28caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:46.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.825+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d4120690 con 0x7f28d41470b0 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28caffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28d8ee4640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28d8ee4640 10 monclient: _renew_subs 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28d8ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d41db370 con 0x7f28d41470b0 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28d8ee4640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28d8ee4640 10 monclient: _renew_subs 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28d8ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:46.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d4146570 con 0x7f28d41470b0 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28d8ee4640 1 librados: init done 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28cc004ce0 con 0x7f28d41470b0 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f28cc0050b0 con 0x7f28d41470b0 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.826+0000 7f28caffd640 10 monclient: dump: 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:46.815 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:46.816 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:46.816 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:46.816 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:46.816 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:46.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.827+0000 7f28d0d99640 4 set_mon_vals no callback set 2024-03-29T11:00:46.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.827+0000 7f28d8ee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:46.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.828+0000 7f28d8ee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:46.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.828+0000 7f28d8ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:46.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.828+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b8005080 con 0x7f28d41470b0 2024-03-29T11:00:46.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.829+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f28cc005210 con 0x7f28d41470b0 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.830+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f28cc084c80 con 0x7f28d41470b0 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.830+0000 7f28caffd640 10 client.4361.objecter ms_dispatch 0x7f28d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.830+0000 7f28caffd640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.830+0000 7f28caffd640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.830+0000 7f28caffd640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.831+0000 7f28d259c640 10 monclient: get_auth_request con 0x7f28b406db10 auth_method 0 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.831+0000 7f28caffd640 10 client.4361.objecter ms_handle_connect 0x7f28b406db10 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.837+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28cc050560 con 0x7f28d41470b0 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.837+0000 7f28caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:46.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:46.837+0000 7f28caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:47.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.154+0000 7f28d8ee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:47.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.154+0000 7f28d8ee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:47.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.154+0000 7f28d8ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:47.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.154+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28b8005330 con 0x7f28d41470b0 2024-03-29T11:00:47.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.155+0000 7f28caffd640 1 -- 172.21.15.107:0/1106419199 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f28cc052490 con 0x7f28d41470b0 2024-03-29T11:00:47.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.155+0000 7f28caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:47.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.155+0000 7f28caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:47.135 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:00:47.135 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:47.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 10 librados: watch_flush enter 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 10 librados: watch_flush exit 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 10 client.4361.objecter successfully canceled tick 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 >> v1:172.21.15.107:6816/882194755 conn(0x7f28b406db10 legacy=0x7f28b406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 10 monclient: shutdown 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 >> v1:172.21.15.107:6790/0 conn(0x7f28d41470b0 legacy=0x7f28d41e6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 shutdown_connections 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 >> 172.21.15.107:0/1106419199 conn(0x7f28d4066a00 msgr2=0x7f28d41d1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 shutdown_connections 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 1 -- 172.21.15.107:0/1106419199 wait complete. 2024-03-29T11:00:47.139 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:47.158+0000 7f28d8ee4640 1 librados: shutdown 2024-03-29T11:00:47.153 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:53.960 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:54.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.109+0000 7f975a2b1640 10 monclient: get_monmap_and_config 2024-03-29T11:00:54.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.109+0000 7f975a2b1640 10 monclient: build_initial_monmap 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.109+0000 7f975a2b1640 10 monclient: monmap: 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:00:54.110485+0000 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:00:54.110485+0000 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:00:54.090 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:54.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.110+0000 7f975a2b1640 1 Processor -- start 2024-03-29T11:00:54.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.110+0000 7f975a2b1640 1 -- start start 2024-03-29T11:00:54.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.110+0000 7f975a2b1640 10 monclient: init 2024-03-29T11:00:54.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f975a2b1640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:54.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f975a2b1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:54.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f975a2b1640 10 monclient(hunting): picked mon.noname-c con 0x7f97540a9750 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f975a2b1640 10 monclient(hunting): picked mon.noname-a con 0x7f97540aad20 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f975a2b1640 10 monclient(hunting): picked mon.noname-b con 0x7f97540a9b30 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f9759ab0640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f97540a9b30 0x7f975414a040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46542/0 (socket says 172.21.15.107:46542) 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f9759ab0640 1 -- 172.21.15.107:0/1130909947 learned_addr learned my addr 172.21.15.107:0/1130909947 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f97592af640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f97540a9750 0x7f97540ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:58816/0 (socket says 172.21.15.107:58816) 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f975a2b1640 1 -- 172.21.15.107:0/1130909947 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540af4d0 con 0x7f97540aad20 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f975a2b1640 1 -- 172.21.15.107:0/1130909947 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540af7b0 con 0x7f97540a9750 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.111+0000 7f975a2b1640 1 -- 172.21.15.107:0/1130909947 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540acab0 con 0x7f97540a9b30 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f975a2b1640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f975a2b1640 10 monclient(hunting): authenticate will time out at 985.380096s 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006641745 0 0) 0x7f97540acab0 con 0x7f97540a9b30 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:54.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9740004630 con 0x7f97540a9b30 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504154819 0 0) 0x7f97540af7b0 con 0x7f97540a9750 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.112+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97540acab0 con 0x7f97540a9750 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.113+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3906868343 0 0) 0x7f9740004630 con 0x7f97540a9b30 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.113+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 >> v1:172.21.15.129:6789/0 conn(0x7f97540a9750 legacy=0x7f97540ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.113+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 >> v1:172.21.15.107:6789/0 conn(0x7f97540aad20 legacy=0x7f9754146910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:54.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 1 monclient: found mon.noname-b 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97540aba30 con 0x7f97540a9b30 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: _finish_auth 0 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f974c001a10 con 0x7f97540a9b30 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f975a2b1640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f975a2b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: dump: 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f974c001bf0 con 0x7f97540a9b30 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:54.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 1 -- 172.21.15.107:0/1130909947 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f974c001f80 con 0x7f97540a9b30 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f974bfff640 10 monclient: dump: 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:54.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f975a2b1640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.115+0000 7f975a2b1640 4 set_mon_vals no callback set 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.116+0000 7f975a2b1640 10 monclient: shutdown 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.116+0000 7f975a2b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.116+0000 7f975a2b1640 1 -- 172.21.15.107:0/1130909947 >> v1:172.21.15.107:6790/0 conn(0x7f97540a9b30 legacy=0x7f975414a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.116+0000 7f975a2b1640 1 -- 172.21.15.107:0/1130909947 shutdown_connections 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.116+0000 7f975a2b1640 1 -- 172.21.15.107:0/1130909947 >> 172.21.15.107:0/1130909947 conn(0x7f97540673e0 msgr2=0x7f97540b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.117+0000 7f975a2b1640 1 -- 172.21.15.107:0/1130909947 shutdown_connections 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.117+0000 7f975a2b1640 1 -- 172.21.15.107:0/1130909947 wait complete. 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.117+0000 7f975a2b1640 10 monclient: build_initial_monmap 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.118+0000 7f975a2b1640 10 monclient: monmap: 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:54.105 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.118+0000 7f975a2b1640 1 librados: starting msgr at 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.118+0000 7f975a2b1640 1 librados: starting objecter 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.118+0000 7f975a2b1640 1 Processor -- start 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.118+0000 7f975a2b1640 1 -- start start 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.118+0000 7f975a2b1640 1 librados: setting wanted keys 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.118+0000 7f975a2b1640 1 librados: calling monclient init 2024-03-29T11:00:54.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.118+0000 7f975a2b1640 10 monclient: init 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f975a2b1640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f975a2b1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f975a2b1640 10 monclient(hunting): picked mon.noname-a con 0x7f97540a9750 addr v1:172.21.15.107:6789/0 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f975a2b1640 10 monclient(hunting): picked mon.noname-c con 0x7f97540a9b30 addr v1:172.21.15.107:6790/0 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f9758aae640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f97540a9b30 0x7f975406f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46732/0 (socket says 172.21.15.107:46732) 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f9758aae640 1 -- 172.21.15.107:0/501704050 learned_addr learned my addr 172.21.15.107:0/501704050 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f975a2b1640 10 monclient(hunting): picked mon.noname-b con 0x7f97540aad20 addr v1:172.21.15.129:6789/0 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f9749ffb640 10 client.?.objecter ms_handle_connect 0x7f97540a9b30 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f9749ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9754070ca0 con 0x7f97540a9750 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540714d0 con 0x7f97540aad20 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.119+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975413b2b0 con 0x7f97540a9b30 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.120+0000 7f975a2b1640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:54.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.120+0000 7f975a2b1640 10 monclient(hunting): authenticate will time out at 985.388120s 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.120+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495520357 0 0) 0x7f975413b2b0 con 0x7f97540a9b30 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.120+0000 7f9749ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.120+0000 7f9749ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.120+0000 7f9749ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.120+0000 7f9749ffb640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.120+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9738004eb0 con 0x7f97540a9b30 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.120+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3933770350 0 0) 0x7f9738004eb0 con 0x7f97540a9b30 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.121+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f975413b2b0 con 0x7f97540a9b30 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.121+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9744003fe0 con 0x7f97540a9b30 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.121+0000 7f9749ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.121+0000 7f9749ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.121+0000 7f9749ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:54.108 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.121+0000 7f9749ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.121+0000 7f9749ffb640 10 client.?.objecter ms_handle_connect 0x7f97540aad20 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.121+0000 7f9749ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.121+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4260940915 0 0) 0x7f975413b2b0 con 0x7f97540a9b30 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 >> v1:172.21.15.129:6789/0 conn(0x7f97540aad20 legacy=0x7f97541e5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 >> v1:172.21.15.107:6789/0 conn(0x7f97540a9750 legacy=0x7f97540719f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:54.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:54.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 1 monclient: found mon.c 2024-03-29T11:00:54.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:54.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:54.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975406e8b0 con 0x7f97540a9b30 2024-03-29T11:00:54.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f9749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f975a2b1640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f975a2b1640 10 monclient: _renew_subs 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f975a2b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.122+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975406eb60 con 0x7f97540a9b30 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.123+0000 7f975a2b1640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.123+0000 7f975a2b1640 10 monclient: _renew_subs 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.123+0000 7f975a2b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.123+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f975406fcd0 con 0x7f97540a9b30 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.123+0000 7f975a2b1640 1 librados: init done 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.123+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97440041b0 con 0x7f97540a9b30 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.123+0000 7f9749ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:54.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.123+0000 7f974b7fe640 4 set_mon_vals no callback set 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.124+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9744001180 con 0x7f97540a9b30 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.124+0000 7f9749ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.124+0000 7f9749ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.124+0000 7f9749ffb640 10 monclient: dump: 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:00:54.113 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.125+0000 7f975a2b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.125+0000 7f975a2b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.125+0000 7f975a2b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.125+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9734005080 con 0x7f97540a9b30 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.126+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f97440012e0 con 0x7f97540a9b30 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.127+0000 7f97592af640 10 monclient: get_auth_request con 0x7f973806db10 auth_method 0 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.127+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f9744084a10 con 0x7f97540a9b30 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.127+0000 7f9749ffb640 10 client.4367.objecter ms_dispatch 0x7f9754000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.127+0000 7f9749ffb640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.127+0000 7f9749ffb640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.127+0000 7f9749ffb640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-29T11:00:54.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.127+0000 7f9749ffb640 10 client.4367.objecter ms_handle_connect 0x7f973806db10 2024-03-29T11:00:54.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.134+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9744050120 con 0x7f97540a9b30 2024-03-29T11:00:54.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.134+0000 7f9749ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:54.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.134+0000 7f9749ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:54.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.429+0000 7f975a2b1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:54.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.429+0000 7f975a2b1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:54.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.429+0000 7f975a2b1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:00:54.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.429+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9734005330 con 0x7f97540a9b30 2024-03-29T11:00:54.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.430+0000 7f9749ffb640 1 -- 172.21.15.107:0/501704050 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f9744052220 con 0x7f97540a9b30 2024-03-29T11:00:54.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.430+0000 7f9749ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:54.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.430+0000 7f9749ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:54.411 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:00:54.411 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:54.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 10 librados: watch_flush enter 2024-03-29T11:00:54.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 10 librados: watch_flush exit 2024-03-29T11:00:54.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-29T11:00:54.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 10 client.4367.objecter successfully canceled tick 2024-03-29T11:00:54.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 >> v1:172.21.15.107:6816/882194755 conn(0x7f973806db10 legacy=0x7f973806ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.413 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 10 monclient: shutdown 2024-03-29T11:00:54.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:54.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 >> v1:172.21.15.107:6790/0 conn(0x7f97540a9b30 legacy=0x7f975406f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:54.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 shutdown_connections 2024-03-29T11:00:54.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 >> 172.21.15.107:0/501704050 conn(0x7f97540673e0 msgr2=0x7f9754069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:54.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 shutdown_connections 2024-03-29T11:00:54.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 1 -- 172.21.15.107:0/501704050 wait complete. 2024-03-29T11:00:54.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:00:54.433+0000 7f975a2b1640 1 librados: shutdown 2024-03-29T11:00:54.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:01.234 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.401+0000 7fabcad6c640 10 monclient: get_monmap_and_config 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.401+0000 7fabcad6c640 10 monclient: build_initial_monmap 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.401+0000 7fabcad6c640 10 monclient: monmap: 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:01.383 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:01:01.402730+0000 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:01:01.402730+0000 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:01:01.384 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:01:01.385 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:01.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.402+0000 7fabcad6c640 1 Processor -- start 2024-03-29T11:01:01.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.402+0000 7fabcad6c640 1 -- start start 2024-03-29T11:01:01.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.402+0000 7fabcad6c640 10 monclient: init 2024-03-29T11:01:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 10 monclient(hunting): picked mon.noname-c con 0x7fabc40b4100 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:01.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 10 monclient(hunting): picked mon.noname-b con 0x7fabc413ee90 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:01.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 10 monclient(hunting): picked mon.noname-a con 0x7fabc40af900 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:01.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40afd10 con 0x7fabc40af900 2024-03-29T11:01:01.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40af6f0 con 0x7fabc40b4100 2024-03-29T11:01:01.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40aaf60 con 0x7fabc413ee90 2024-03-29T11:01:01.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:01.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.403+0000 7fabcad6c640 10 monclient(hunting): authenticate will time out at 992.671792s 2024-03-29T11:01:01.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.407+0000 7fabc3fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fabc413ee90 0x7fabc40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46628/0 (socket says 172.21.15.107:46628) 2024-03-29T11:01:01.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.407+0000 7fabc3fff640 1 -- 172.21.15.107:0/4061802996 learned_addr learned my addr 172.21.15.107:0/4061802996 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.407+0000 7fabc8b08640 1 --1- 172.21.15.107:0/4061802996 >> v1:172.21.15.129:6789/0 conn(0x7fabc40b4100 0x7fabc40a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:34739/0 (socket says 172.21.15.107:34739) 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700949681 0 0) 0x7fabc40aaf60 con 0x7fabc413ee90 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac003520 con 0x7fabc413ee90 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777192796 0 0) 0x7fabc40af6f0 con 0x7fabc40b4100 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc40aaf60 con 0x7fabc40b4100 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770727121 0 0) 0x7fabc40afd10 con 0x7fabc40af900 2024-03-29T11:01:01.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.408+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc40af6f0 con 0x7fabc40af900 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2786056424 0 0) 0x7fabc40af6f0 con 0x7fabc40af900 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 >> v1:172.21.15.107:6790/0 conn(0x7fabc413ee90 legacy=0x7fabc40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 >> v1:172.21.15.129:6789/0 conn(0x7fabc40b4100 legacy=0x7fabc40a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 1 monclient: found mon.noname-a 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc413e970 con 0x7fabc40af900 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:01.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fabbc0017e0 con 0x7fabc40af900 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabcad6c640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabcad6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: dump: 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.401 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabcad6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabbc001a00 con 0x7fabc40af900 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 1 -- 172.21.15.107:0/4061802996 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fabbc001d90 con 0x7fabc40af900 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabc37fe640 10 monclient: dump: 2024-03-29T11:01:01.402 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabcad6c640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.409+0000 7fabcad6c640 4 set_mon_vals no callback set 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.410+0000 7fabcad6c640 10 monclient: shutdown 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.410+0000 7fabcad6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.410+0000 7fabcad6c640 1 -- 172.21.15.107:0/4061802996 >> v1:172.21.15.107:6789/0 conn(0x7fabc40af900 legacy=0x7fabc413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.411+0000 7fabcad6c640 1 -- 172.21.15.107:0/4061802996 shutdown_connections 2024-03-29T11:01:01.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.411+0000 7fabcad6c640 1 -- 172.21.15.107:0/4061802996 >> 172.21.15.107:0/4061802996 conn(0x7fabc40b35e0 msgr2=0x7fabc40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.411+0000 7fabcad6c640 1 -- 172.21.15.107:0/4061802996 shutdown_connections 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.411+0000 7fabcad6c640 1 -- 172.21.15.107:0/4061802996 wait complete. 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.412+0000 7fabcad6c640 10 monclient: build_initial_monmap 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.412+0000 7fabcad6c640 10 monclient: monmap: 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.413+0000 7fabcad6c640 1 librados: starting msgr at 2024-03-29T11:01:01.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.413+0000 7fabcad6c640 1 librados: starting objecter 2024-03-29T11:01:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.413+0000 7fabcad6c640 1 Processor -- start 2024-03-29T11:01:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.413+0000 7fabcad6c640 1 -- start start 2024-03-29T11:01:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.413+0000 7fabcad6c640 1 librados: setting wanted keys 2024-03-29T11:01:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.413+0000 7fabcad6c640 1 librados: calling monclient init 2024-03-29T11:01:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.413+0000 7fabcad6c640 10 monclient: init 2024-03-29T11:01:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.415+0000 7fabcad6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.415+0000 7fabcad6c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:01.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.415+0000 7fabcad6c640 10 monclient(hunting): picked mon.noname-b con 0x7fabc40af900 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:01.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabcad6c640 10 monclient(hunting): picked mon.noname-a con 0x7fabc40b4100 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:01.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabcad6c640 10 monclient(hunting): picked mon.noname-c con 0x7fabc413ee90 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:01.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabcad6c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc41dfe60 con 0x7fabc40b4100 2024-03-29T11:01:01.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabcad6c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc4150960 con 0x7fabc40af900 2024-03-29T11:01:01.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabcad6c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc4150c40 con 0x7fabc413ee90 2024-03-29T11:01:01.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabcad6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:01.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabcad6c640 10 monclient(hunting): authenticate will time out at 992.687074s 2024-03-29T11:01:01.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabc3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fabc40b4100 0x7fabc41df040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54041/0 (socket says 172.21.15.107:54041) 2024-03-29T11:01:01.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabc3fff640 1 -- 172.21.15.107:0/3141575240 learned_addr learned my addr 172.21.15.107:0/3141575240 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:01.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.419+0000 7fabc17fa640 10 client.?.objecter ms_handle_connect 0x7fabc40b4100 2024-03-29T11:01:01.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:01.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 client.?.objecter ms_handle_connect 0x7fabc413ee90 2024-03-29T11:01:01.416 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:01.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373074023 0 0) 0x7fabc41dfe60 con 0x7fabc40b4100 2024-03-29T11:01:01.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:01:01.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb0004eb0 con 0x7fabc40b4100 2024-03-29T11:01:01.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944646104 0 0) 0x7fabc4150c40 con 0x7fabc413ee90 2024-03-29T11:01:01.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:01:01.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.420+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc41dfe60 con 0x7fabc413ee90 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.423+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4174050144 0 0) 0x7fabb0004eb0 con 0x7fabc40b4100 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.423+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabc4150c40 con 0x7fabc40b4100 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.423+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fabb4001840 con 0x7fabc40b4100 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.423+0000 7fabc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.423+0000 7fabc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.423+0000 7fabc17fa640 10 monclient(hunting): dump: 2024-03-29T11:01:01.418 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.423+0000 7fabc17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.423+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2300940139 0 0) 0x7fabc41dfe60 con 0x7fabc413ee90 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabb0004eb0 con 0x7fabc413ee90 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 10 client.?.objecter ms_handle_connect 0x7fabc40af900 2024-03-29T11:01:01.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fabbc001580 con 0x7fabc413ee90 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 10 monclient(hunting): dump: 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:01.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3051393315 0 0) 0x7fabb0004eb0 con 0x7fabc413ee90 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 >> v1:172.21.15.129:6789/0 conn(0x7fabc40af900 legacy=0x7fabc4151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 >> v1:172.21.15.107:6789/0 conn(0x7fabc40b4100 legacy=0x7fabc41df040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 1 monclient: found mon.c 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc4150f20 con 0x7fabc413ee90 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 10 monclient: _finish_auth 0 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.424+0000 7fabc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabbc001820 con 0x7fabc413ee90 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabc17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabcad6c640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:01:01.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabcad6c640 10 monclient: _renew_subs 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabcad6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc414f1d0 con 0x7fabc413ee90 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabcad6c640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabcad6c640 10 monclient: _renew_subs 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabcad6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc414e8c0 con 0x7fabc413ee90 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabcad6c640 1 librados: init done 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabc27fc640 4 set_mon_vals no callback set 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.425+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fabbc003dc0 con 0x7fabc413ee90 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.426+0000 7fabc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:01.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.426+0000 7fabc17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.426+0000 7fabc17fa640 10 monclient: dump: 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.426+0000 7fabcad6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.427+0000 7fabcad6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.427+0000 7fabcad6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:01.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.427+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba4005080 con 0x7fabc413ee90 2024-03-29T11:01:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.429+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fabbc019d90 con 0x7fabc413ee90 2024-03-29T11:01:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.430+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fabbc083a40 con 0x7fabc413ee90 2024-03-29T11:01:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.430+0000 7fabc17fa640 10 client.4373.objecter ms_dispatch 0x7fabc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.430+0000 7fabc17fa640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.430+0000 7fabc17fa640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.430+0000 7fabc17fa640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:01:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.431+0000 7fabc8b08640 10 monclient: get_auth_request con 0x7fabb006df90 auth_method 0 2024-03-29T11:01:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.432+0000 7fabc17fa640 10 client.4373.objecter ms_handle_connect 0x7fabb006df90 2024-03-29T11:01:01.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.438+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabbc04f320 con 0x7fabc413ee90 2024-03-29T11:01:01.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.438+0000 7fabc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:01.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.438+0000 7fabc17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:01.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.765+0000 7fabcad6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:01.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.765+0000 7fabcad6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:01.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.765+0000 7fabcad6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:01.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.765+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faba4005330 con 0x7fabc413ee90 2024-03-29T11:01:01.746 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:01:01.746 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:01.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.766+0000 7fabc17fa640 1 -- 172.21.15.107:0/3141575240 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fabbc051250 con 0x7fabc413ee90 2024-03-29T11:01:01.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.766+0000 7fabc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:01.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.766+0000 7fabc17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:01.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.769+0000 7fabcad6c640 10 librados: watch_flush enter 2024-03-29T11:01:01.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.769+0000 7fabcad6c640 10 librados: watch_flush exit 2024-03-29T11:01:01.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.769+0000 7fabcad6c640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:01:01.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.769+0000 7fabcad6c640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:01:01.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.769+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 >> v1:172.21.15.107:6816/882194755 conn(0x7fabb006df90 legacy=0x7fabb0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.769+0000 7fabcad6c640 10 monclient: shutdown 2024-03-29T11:01:01.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.770+0000 7fabcad6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:01.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.770+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 >> v1:172.21.15.107:6790/0 conn(0x7fabc413ee90 legacy=0x7fabc41df750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:01.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.770+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 shutdown_connections 2024-03-29T11:01:01.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.770+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 >> 172.21.15.107:0/3141575240 conn(0x7fabc40b35e0 msgr2=0x7fabc40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:01.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.772+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 shutdown_connections 2024-03-29T11:01:01.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.773+0000 7fabcad6c640 1 -- 172.21.15.107:0/3141575240 wait complete. 2024-03-29T11:01:01.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:01.773+0000 7fabcad6c640 1 librados: shutdown 2024-03-29T11:01:01.770 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:08.577 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.746+0000 7f1069e12640 10 monclient: get_monmap_and_config 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.746+0000 7f1069e12640 10 monclient: build_initial_monmap 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.746+0000 7f1069e12640 10 monclient: monmap: 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:01:08.747906+0000 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:01:08.747906+0000 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:08.727 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.748+0000 7f1069e12640 1 Processor -- start 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.748+0000 7f1069e12640 1 -- start start 2024-03-29T11:01:08.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.748+0000 7f1069e12640 10 monclient: init 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.749+0000 7f1069e12640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.749+0000 7f1069e12640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.749+0000 7f1069e12640 10 monclient(hunting): picked mon.noname-b con 0x7f105c006eb0 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.749+0000 7f1068e10640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f105c006eb0 0x7f105c031760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41168/0 (socket says 172.21.15.107:41168) 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.752+0000 7f1068e10640 1 -- 172.21.15.107:0/2379998062 learned_addr learned my addr 172.21.15.107:0/2379998062 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:08.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.752+0000 7f1069e12640 10 monclient(hunting): picked mon.noname-c con 0x7f105c038f10 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.752+0000 7f1069e12640 10 monclient(hunting): picked mon.noname-a con 0x7f105c036690 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.752+0000 7f1069e12640 1 -- 172.21.15.107:0/2379998062 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c033c90 con 0x7f105c036690 2024-03-29T11:01:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f1069e12640 1 -- 172.21.15.107:0/2379998062 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c0389f0 con 0x7f105c038f10 2024-03-29T11:01:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f1069e12640 1 -- 172.21.15.107:0/2379998062 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c030890 con 0x7f105c006eb0 2024-03-29T11:01:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f1069e12640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f1069e12640 10 monclient(hunting): authenticate will time out at 1000.021541s 2024-03-29T11:01:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f10637fe640 1 -- 172.21.15.107:0/2379998062 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563174396 0 0) 0x7f105c0389f0 con 0x7f105c038f10 2024-03-29T11:01:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f10637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:08.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f10637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f10637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f10637fe640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.753+0000 7f10637fe640 1 -- 172.21.15.107:0/2379998062 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1054003520 con 0x7f105c038f10 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.754+0000 7f10637fe640 1 -- 172.21.15.107:0/2379998062 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648229624 0 0) 0x7f1054003520 con 0x7f105c038f10 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.754+0000 7f10637fe640 1 -- 172.21.15.107:0/2379998062 >> v1:172.21.15.107:6790/0 conn(0x7f105c006eb0 legacy=0x7f105c031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.754+0000 7f10637fe640 1 -- 172.21.15.107:0/2379998062 >> v1:172.21.15.107:6789/0 conn(0x7f105c036690 legacy=0x7f105c036b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 1 monclient: found mon.noname-c 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 1 -- 172.21.15.107:0/2379998062 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c0cbc40 con 0x7f105c038f10 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:08.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 1 -- 172.21.15.107:0/2379998062 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1058001690 con 0x7f105c038f10 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f10637fe640 10 monclient: dump: 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:08.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f1069e12640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.756+0000 7f1069e12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.757+0000 7f10637fe640 1 -- 172.21.15.107:0/2379998062 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10580018f0 con 0x7f105c038f10 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.757+0000 7f10637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.757+0000 7f10637fe640 1 -- 172.21.15.107:0/2379998062 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1058001d00 con 0x7f105c038f10 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.757+0000 7f10637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.757+0000 7f10637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.757+0000 7f10637fe640 10 monclient: dump: 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.757+0000 7f1069e12640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.757+0000 7f1069e12640 4 set_mon_vals no callback set 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.758+0000 7f1069e12640 10 monclient: shutdown 2024-03-29T11:01:08.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.758+0000 7f1069e12640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.758+0000 7f1069e12640 1 -- 172.21.15.107:0/2379998062 >> v1:172.21.15.129:6789/0 conn(0x7f105c038f10 legacy=0x7f105c039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.760+0000 7f1069e12640 1 -- 172.21.15.107:0/2379998062 shutdown_connections 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.760+0000 7f1069e12640 1 -- 172.21.15.107:0/2379998062 >> 172.21.15.107:0/2379998062 conn(0x7f105c005c20 msgr2=0x7f105c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.760+0000 7f1069e12640 1 -- 172.21.15.107:0/2379998062 shutdown_connections 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.760+0000 7f1069e12640 1 -- 172.21.15.107:0/2379998062 wait complete. 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.761+0000 7f1069e12640 10 monclient: build_initial_monmap 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.761+0000 7f1069e12640 10 monclient: monmap: 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:01:08.742 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:01:08.743 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:08.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.762+0000 7f1069e12640 1 librados: starting msgr at 2024-03-29T11:01:08.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.762+0000 7f1069e12640 1 librados: starting objecter 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.762+0000 7f1069e12640 1 Processor -- start 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.762+0000 7f1069e12640 1 -- start start 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.762+0000 7f1069e12640 1 librados: setting wanted keys 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.762+0000 7f1069e12640 1 librados: calling monclient init 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.762+0000 7f1069e12640 10 monclient: init 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.763+0000 7f1069e12640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.763+0000 7f1069e12640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.763+0000 7f1069e12640 10 monclient(hunting): picked mon.noname-c con 0x7f105c006eb0 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.763+0000 7f1068e10640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f105c006eb0 0x7f105c164920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41654/0 (socket says 172.21.15.107:41654) 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.763+0000 7f1068e10640 1 -- 172.21.15.107:0/1895347941 learned_addr learned my addr 172.21.15.107:0/1895347941 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.763+0000 7f10617fa640 10 client.?.objecter ms_handle_connect 0x7f105c006eb0 2024-03-29T11:01:08.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.763+0000 7f10617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:08.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.763+0000 7f1069e12640 10 monclient(hunting): picked mon.noname-a con 0x7f105c036690 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:08.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.763+0000 7f1069e12640 10 monclient(hunting): picked mon.noname-b con 0x7f105c038f10 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:08.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.766+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c164400 con 0x7f105c036690 2024-03-29T11:01:08.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.766+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c165030 con 0x7f105c038f10 2024-03-29T11:01:08.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.766+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105c15e6f0 con 0x7f105c006eb0 2024-03-29T11:01:08.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.766+0000 7f1069e12640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:08.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.766+0000 7f1069e12640 10 monclient(hunting): authenticate will time out at 1000.034760s 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 client.?.objecter ms_handle_connect 0x7f105c038f10 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893169180 0 0) 0x7f105c15e6f0 con 0x7f105c006eb0 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 monclient(hunting): _negotiate 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1048004eb0 con 0x7f105c006eb0 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692215967 0 0) 0x7f105c165030 con 0x7f105c038f10 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 monclient(hunting): _negotiate 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:01:08.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.767+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f105c15e6f0 con 0x7f105c038f10 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2569919674 0 0) 0x7f1048004eb0 con 0x7f105c006eb0 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f105c165030 con 0x7f105c006eb0 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f106404fdd0 con 0x7f105c006eb0 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient(hunting): dump: 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:08.750 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1421674996 0 0) 0x7f105c165030 con 0x7f105c006eb0 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 >> v1:172.21.15.129:6789/0 conn(0x7f105c038f10 legacy=0x7f105c15dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 >> v1:172.21.15.107:6789/0 conn(0x7f105c036690 legacy=0x7f105c15d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient: _finish_hunting 0 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 monclient: found mon.c 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c15e880 con 0x7f105c006eb0 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient: _finish_auth 0 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient: _check_auth_tickets 2024-03-29T11:01:08.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f106404f400 con 0x7f105c006eb0 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1064075f10 con 0x7f105c006eb0 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.768+0000 7f10617fa640 10 monclient: dump: 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:08.752 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1062ffd640 4 set_mon_vals no callback set 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1069e12640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1069e12640 10 monclient: _renew_subs 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1069e12640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105c0cfd20 con 0x7f105c006eb0 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1069e12640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1069e12640 10 monclient: _renew_subs 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1069e12640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f105c0cf460 con 0x7f105c006eb0 2024-03-29T11:01:08.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.769+0000 7f1069e12640 1 librados: init done 2024-03-29T11:01:08.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.770+0000 7f1069e12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:08.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.770+0000 7f1069e12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:08.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.770+0000 7f1069e12640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:08.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.771+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f1064067bf0 con 0x7f105c006eb0 2024-03-29T11:01:08.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.770+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105c007b90 con 0x7f105c006eb0 2024-03-29T11:01:08.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.772+0000 7f1063fff640 10 monclient: get_auth_request con 0x7f104807eb70 auth_method 0 2024-03-29T11:01:08.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.772+0000 7f10617fa640 10 client.4376.objecter ms_handle_connect 0x7f104807eb70 2024-03-29T11:01:08.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.772+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f10640ff580 con 0x7f105c006eb0 2024-03-29T11:01:08.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.772+0000 7f10617fa640 10 client.4376.objecter ms_dispatch 0x7f1064000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:08.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.772+0000 7f10617fa640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:08.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.772+0000 7f10617fa640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:08.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.772+0000 7f10617fa640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T11:01:08.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.781+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10640cae60 con 0x7f105c006eb0 2024-03-29T11:01:08.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.781+0000 7f10617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:08.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:08.781+0000 7f10617fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:09.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.105+0000 7f1069e12640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:09.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.105+0000 7f1069e12640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:09.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.105+0000 7f1069e12640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:09.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.105+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f105c1641f0 con 0x7f105c006eb0 2024-03-29T11:01:09.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.109+0000 7f10617fa640 1 -- 172.21.15.107:0/1895347941 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f10640ccd90 con 0x7f105c006eb0 2024-03-29T11:01:09.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.109+0000 7f10617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:09.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.109+0000 7f10617fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:09.090 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:01:09.090 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:09.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.113+0000 7f1069e12640 10 librados: watch_flush enter 2024-03-29T11:01:09.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.113+0000 7f1069e12640 10 librados: watch_flush exit 2024-03-29T11:01:09.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.113+0000 7f1069e12640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T11:01:09.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.113+0000 7f1069e12640 10 client.4376.objecter successfully canceled tick 2024-03-29T11:01:09.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.114+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 >> v1:172.21.15.107:6816/882194755 conn(0x7f104807eb70 legacy=0x7f1048081030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:09.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.114+0000 7f1069e12640 10 monclient: shutdown 2024-03-29T11:01:09.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.114+0000 7f1069e12640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:09.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.114+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 >> v1:172.21.15.107:6790/0 conn(0x7f105c006eb0 legacy=0x7f105c164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:09.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.117+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 shutdown_connections 2024-03-29T11:01:09.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.117+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 >> 172.21.15.107:0/1895347941 conn(0x7f105c005c20 msgr2=0x7f105c03c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:09.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.117+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 shutdown_connections 2024-03-29T11:01:09.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.118+0000 7f1069e12640 1 -- 172.21.15.107:0/1895347941 wait complete. 2024-03-29T11:01:09.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:09.118+0000 7f1069e12640 1 librados: shutdown 2024-03-29T11:01:09.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:15.922 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.091+0000 7ff3215c6640 10 monclient: get_monmap_and_config 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.091+0000 7ff3215c6640 10 monclient: build_initial_monmap 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.091+0000 7ff3215c6640 10 monclient: monmap: 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:16.071 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:01:16.092561+0000 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:01:16.092561+0000 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:16.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.092+0000 7ff3215c6640 1 Processor -- start 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.092+0000 7ff3215c6640 1 -- start start 2024-03-29T11:01:16.073 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.092+0000 7ff3215c6640 10 monclient: init 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff31c1457f0 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff31c13f650 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff31c0a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:16.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c0aaef0 con 0x7ff31c0a9d70 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c145bd0 con 0x7ff31c13f650 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c145eb0 con 0x7ff31c1457f0 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.093+0000 7ff3215c6640 10 monclient(hunting): authenticate will time out at 1007.361701s 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.094+0000 7ff31affd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff31c1457f0 0x7ff31c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38446/0 (socket says 172.21.15.107:38446) 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.094+0000 7ff31affd640 1 -- 172.21.15.107:0/2050596136 learned_addr learned my addr 172.21.15.107:0/2050596136 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.095+0000 7ff319ffb640 1 -- 172.21.15.107:0/2050596136 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611568356 0 0) 0x7ff31c145eb0 con 0x7ff31c1457f0 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.095+0000 7ff319ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.095+0000 7ff319ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.095+0000 7ff319ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.095+0000 7ff319ffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.095+0000 7ff319ffb640 1 -- 172.21.15.107:0/2050596136 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff304003520 con 0x7ff31c1457f0 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.095+0000 7ff319ffb640 1 -- 172.21.15.107:0/2050596136 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566249726 0 0) 0x7ff304003520 con 0x7ff31c1457f0 2024-03-29T11:01:16.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.095+0000 7ff319ffb640 1 -- 172.21.15.107:0/2050596136 >> v1:172.21.15.129:6789/0 conn(0x7ff31c13f650 legacy=0x7ff31c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:16.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 1 -- 172.21.15.107:0/2050596136 >> v1:172.21.15.107:6789/0 conn(0x7ff31c0a9d70 legacy=0x7ff31c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:16.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:16.078 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 1 monclient: found mon.noname-b 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 1 -- 172.21.15.107:0/2050596136 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c146340 con 0x7ff31c1457f0 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 1 -- 172.21.15.107:0/2050596136 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff30c0019f0 con 0x7ff31c1457f0 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.096+0000 7ff319ffb640 10 monclient: dump: 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:16.079 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.097+0000 7ff3215c6640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.097+0000 7ff3215c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.097+0000 7ff319ffb640 1 -- 172.21.15.107:0/2050596136 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff30c001c50 con 0x7ff31c1457f0 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.097+0000 7ff319ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.097+0000 7ff3215c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.097+0000 7ff3215c6640 4 set_mon_vals no callback set 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.098+0000 7ff3215c6640 10 monclient: shutdown 2024-03-29T11:01:16.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.098+0000 7ff3215c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:16.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.098+0000 7ff3215c6640 1 -- 172.21.15.107:0/2050596136 >> v1:172.21.15.107:6790/0 conn(0x7ff31c1457f0 legacy=0x7ff31c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.098+0000 7ff319ffb640 1 -- 172.21.15.107:0/2050596136 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff30c0020a0 con 0x7ff31c1457f0 2024-03-29T11:01:16.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.098+0000 7ff319ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:01:16.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.098+0000 7ff3215c6640 1 -- 172.21.15.107:0/2050596136 shutdown_connections 2024-03-29T11:01:16.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.098+0000 7ff3215c6640 1 -- 172.21.15.107:0/2050596136 >> 172.21.15.107:0/2050596136 conn(0x7ff31c066a00 msgr2=0x7ff31c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:16.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.098+0000 7ff3215c6640 1 -- 172.21.15.107:0/2050596136 shutdown_connections 2024-03-29T11:01:16.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.098+0000 7ff3215c6640 1 -- 172.21.15.107:0/2050596136 wait complete. 2024-03-29T11:01:16.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.099+0000 7ff3215c6640 10 monclient: build_initial_monmap 2024-03-29T11:01:16.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.099+0000 7ff3215c6640 10 monclient: monmap: 2024-03-29T11:01:16.083 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:16.083 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:16.083 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:01:16.083 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:01:16.083 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.100+0000 7ff3215c6640 1 librados: starting msgr at 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.100+0000 7ff3215c6640 1 librados: starting objecter 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.101+0000 7ff3215c6640 1 Processor -- start 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.101+0000 7ff3215c6640 1 -- start start 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.101+0000 7ff3215c6640 1 librados: setting wanted keys 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.101+0000 7ff3215c6640 1 librados: calling monclient init 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.101+0000 7ff3215c6640 10 monclient: init 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:16.084 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff31c0a9d70 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff31c13f650 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff31c1457f0 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c142380 con 0x7ff31c13f650 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c1dae30 con 0x7ff31c1457f0 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c135770 con 0x7ff31c0a9d70 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:16.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.102+0000 7ff3215c6640 10 monclient(hunting): authenticate will time out at 1007.370957s 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.104+0000 7ff31affd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff31c0a9d70 0x7ff31c1da720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40184/0 (socket says 172.21.15.107:40184) 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.104+0000 7ff31affd640 1 -- 172.21.15.107:0/1503432569 learned_addr learned my addr 172.21.15.107:0/1503432569 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.105+0000 7ff3037fe640 10 client.?.objecter ms_handle_connect 0x7ff31c0a9d70 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.105+0000 7ff3037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.106+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245147624 0 0) 0x7ff31c135770 con 0x7ff31c0a9d70 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.106+0000 7ff3037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.106+0000 7ff3037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.106+0000 7ff3037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.106+0000 7ff3037fe640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:01:16.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.106+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff308004eb0 con 0x7ff31c0a9d70 2024-03-29T11:01:16.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.108+0000 7ff3037fe640 10 client.?.objecter ms_handle_connect 0x7ff31c13f650 2024-03-29T11:01:16.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.108+0000 7ff3037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:16.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.108+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541869320 0 0) 0x7ff31c142380 con 0x7ff31c13f650 2024-03-29T11:01:16.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.108+0000 7ff3037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:16.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.108+0000 7ff3037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:16.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.108+0000 7ff3037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:16.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.108+0000 7ff3037fe640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:01:16.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.108+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff31c135770 con 0x7ff31c13f650 2024-03-29T11:01:16.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1217964539 0 0) 0x7ff31c135770 con 0x7ff31c13f650 2024-03-29T11:01:16.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff31c142380 con 0x7ff31c13f650 2024-03-29T11:01:16.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff314001830 con 0x7ff31c13f650 2024-03-29T11:01:16.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient(hunting): dump: 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:16.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1531370112 0 0) 0x7ff308004eb0 con 0x7ff31c0a9d70 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff31c135770 con 0x7ff31c0a9d70 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff30c0019b0 con 0x7ff31c0a9d70 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient(hunting): dump: 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:16.092 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2033830019 0 0) 0x7ff31c135770 con 0x7ff31c0a9d70 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 >> v1:172.21.15.129:6789/0 conn(0x7ff31c1457f0 legacy=0x7ff31c135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 >> v1:172.21.15.107:6789/0 conn(0x7ff31c13f650 legacy=0x7ff31c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 monclient: found mon.c 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c141c80 con 0x7ff31c0a9d70 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:16.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3215c6640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3215c6640 10 monclient: _renew_subs 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.109+0000 7ff3215c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c141f60 con 0x7ff31c0a9d70 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff30c004e10 con 0x7ff31c0a9d70 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff30c005160 con 0x7ff31c0a9d70 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3197fa640 4 set_mon_vals no callback set 2024-03-29T11:01:16.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3037fe640 10 monclient: dump: 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3215c6640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3215c6640 10 monclient: _renew_subs 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3215c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c137800 con 0x7ff31c0a9d70 2024-03-29T11:01:16.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.110+0000 7ff3215c6640 1 librados: init done 2024-03-29T11:01:16.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.112+0000 7ff3215c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:16.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.112+0000 7ff3215c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:16.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.112+0000 7ff3215c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:16.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.112+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31c075f10 con 0x7ff31c0a9d70 2024-03-29T11:01:16.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.113+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7ff30c01b130 con 0x7ff31c0a9d70 2024-03-29T11:01:16.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.114+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7ff30c085020 con 0x7ff31c0a9d70 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.114+0000 7ff3037fe640 10 client.4382.objecter ms_dispatch 0x7ff31c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.114+0000 7ff3037fe640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.114+0000 7ff31a7fc640 10 monclient: get_auth_request con 0x7ff30807ee50 auth_method 0 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.114+0000 7ff3037fe640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.114+0000 7ff3037fe640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T11:01:16.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.114+0000 7ff3037fe640 10 client.4382.objecter ms_handle_connect 0x7ff30807ee50 2024-03-29T11:01:16.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.125+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff30c050900 con 0x7ff31c0a9d70 2024-03-29T11:01:16.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.125+0000 7ff3037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:16.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.125+0000 7ff3037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:16.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.438+0000 7ff3215c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:16.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.438+0000 7ff3215c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:16.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.438+0000 7ff3215c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:16.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.438+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff31c0b3680 con 0x7ff31c0a9d70 2024-03-29T11:01:16.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.440+0000 7ff3037fe640 1 -- 172.21.15.107:0/1503432569 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7ff30c052830 con 0x7ff31c0a9d70 2024-03-29T11:01:16.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.440+0000 7ff3037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:16.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.440+0000 7ff3037fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:16.421 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:01:16.421 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.444+0000 7ff3215c6640 10 librados: watch_flush enter 2024-03-29T11:01:16.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.444+0000 7ff3215c6640 10 librados: watch_flush exit 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.444+0000 7ff3215c6640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.444+0000 7ff3215c6640 10 client.4382.objecter successfully canceled tick 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.444+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 >> v1:172.21.15.107:6816/882194755 conn(0x7ff30807ee50 legacy=0x7ff308081310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.444+0000 7ff3215c6640 10 monclient: shutdown 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.445+0000 7ff3215c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.445+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 >> v1:172.21.15.107:6790/0 conn(0x7ff31c0a9d70 legacy=0x7ff31c1da720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.445+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 shutdown_connections 2024-03-29T11:01:16.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.445+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 >> 172.21.15.107:0/1503432569 conn(0x7ff31c066a00 msgr2=0x7ff31c0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:16.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.447+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 shutdown_connections 2024-03-29T11:01:16.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.448+0000 7ff3215c6640 1 -- 172.21.15.107:0/1503432569 wait complete. 2024-03-29T11:01:16.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:16.448+0000 7ff3215c6640 1 librados: shutdown 2024-03-29T11:01:16.445 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:23.252 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.420+0000 7fb196c15640 10 monclient: get_monmap_and_config 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.420+0000 7fb196c15640 10 monclient: build_initial_monmap 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.420+0000 7fb196c15640 10 monclient: monmap: 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:23.400 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:01:23.421573+0000 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:01:23.421573+0000 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:01:23.401 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:23.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.421+0000 7fb196c15640 1 Processor -- start 2024-03-29T11:01:23.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.421+0000 7fb196c15640 1 -- start start 2024-03-29T11:01:23.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.421+0000 7fb196c15640 10 monclient: init 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.422+0000 7fb196c15640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb196c15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb196c15640 10 monclient(hunting): picked mon.noname-c con 0x7fb1901457f0 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb196c15640 10 monclient(hunting): picked mon.noname-b con 0x7fb19013f650 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:23.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb196c15640 10 monclient(hunting): picked mon.noname-a con 0x7fb1900a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:23.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb196c15640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1900aaef0 con 0x7fb1900a9d70 2024-03-29T11:01:23.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb196c15640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb190145bd0 con 0x7fb1901457f0 2024-03-29T11:01:23.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb196c15640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb190145eb0 con 0x7fb19013f650 2024-03-29T11:01:23.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb196c15640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:23.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb196c15640 10 monclient(hunting): authenticate will time out at 1014.691393s 2024-03-29T11:01:23.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb18ffff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb19013f650 0x7fb1900b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38400/0 (socket says 172.21.15.107:38400) 2024-03-29T11:01:23.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb18ffff640 1 -- 172.21.15.107:0/2990075849 learned_addr learned my addr 172.21.15.107:0/2990075849 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.423+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263542801 0 0) 0x7fb1900aaef0 con 0x7fb1900a9d70 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb180003520 con 0x7fb1900a9d70 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884089381 0 0) 0x7fb190145eb0 con 0x7fb19013f650 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1900aaef0 con 0x7fb19013f650 2024-03-29T11:01:23.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3458222423 0 0) 0x7fb180003520 con 0x7fb1900a9d70 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 >> v1:172.21.15.107:6790/0 conn(0x7fb19013f650 legacy=0x7fb1900b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.424+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 >> v1:172.21.15.129:6789/0 conn(0x7fb1901457f0 legacy=0x7fb19013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190146340 con 0x7fb1900a9d70 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:23.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb188001a10 con 0x7fb1900a9d70 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.426+0000 7fb18f7fe640 10 monclient: dump: 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:23.408 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb196c15640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb196c15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb188001b70 con 0x7fb1900a9d70 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb18f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb18f7fe640 1 -- 172.21.15.107:0/2990075849 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb188001f80 con 0x7fb1900a9d70 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb18f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb18f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb18f7fe640 10 monclient: dump: 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:23.409 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:23.410 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.410 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:23.410 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:23.410 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:23.410 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:23.410 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:23.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb196c15640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:23.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.427+0000 7fb196c15640 4 set_mon_vals no callback set 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.428+0000 7fb196c15640 10 monclient: shutdown 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.428+0000 7fb196c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:23.417 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.428+0000 7fb196c15640 1 -- 172.21.15.107:0/2990075849 >> v1:172.21.15.107:6789/0 conn(0x7fb1900a9d70 legacy=0x7fb19014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.428+0000 7fb196c15640 1 -- 172.21.15.107:0/2990075849 shutdown_connections 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.428+0000 7fb196c15640 1 -- 172.21.15.107:0/2990075849 >> 172.21.15.107:0/2990075849 conn(0x7fb190066a00 msgr2=0x7fb190066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.428+0000 7fb196c15640 1 -- 172.21.15.107:0/2990075849 shutdown_connections 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.429+0000 7fb196c15640 1 -- 172.21.15.107:0/2990075849 wait complete. 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.429+0000 7fb196c15640 10 monclient: build_initial_monmap 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.429+0000 7fb196c15640 10 monclient: monmap: 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:01:23.418 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.430+0000 7fb196c15640 1 librados: starting msgr at 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.430+0000 7fb196c15640 1 librados: starting objecter 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.430+0000 7fb196c15640 1 Processor -- start 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.430+0000 7fb196c15640 1 -- start start 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.430+0000 7fb196c15640 1 librados: setting wanted keys 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.430+0000 7fb196c15640 1 librados: calling monclient init 2024-03-29T11:01:23.419 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.430+0000 7fb196c15640 10 monclient: init 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 10 monclient(hunting): picked mon.noname-b con 0x7fb1900a9d70 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 10 monclient(hunting): picked mon.noname-c con 0x7fb19013f650 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 10 monclient(hunting): picked mon.noname-a con 0x7fb1901457f0 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb190137990 con 0x7fb1901457f0 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1900ac790 con 0x7fb1900a9d70 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb190142080 con 0x7fb19013f650 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.431+0000 7fb196c15640 10 monclient(hunting): authenticate will time out at 1014.699405s 2024-03-29T11:01:23.420 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb1951b2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb1901457f0 0x7fb190137280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49098/0 (socket says 172.21.15.107:49098) 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb1951b2640 1 -- 172.21.15.107:0/2156968134 learned_addr learned my addr 172.21.15.107:0/2156968134 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 10 client.?.objecter ms_handle_connect 0x7fb1901457f0 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815516779 0 0) 0x7fb190137990 con 0x7fb1901457f0 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb174004eb0 con 0x7fb1901457f0 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 10 client.?.objecter ms_handle_connect 0x7fb19013f650 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.432+0000 7fb18d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794652367 0 0) 0x7fb190142080 con 0x7fb19013f650 2024-03-29T11:01:23.421 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190137990 con 0x7fb19013f650 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 194991028 0 0) 0x7fb190137990 con 0x7fb19013f650 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb190142080 con 0x7fb19013f650 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb184001930 con 0x7fb19013f650 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 10 monclient(hunting): dump: 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:23.422 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 871067777 0 0) 0x7fb190142080 con 0x7fb19013f650 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.433+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 >> v1:172.21.15.129:6789/0 conn(0x7fb1900a9d70 legacy=0x7fb1901425d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 >> v1:172.21.15.107:6789/0 conn(0x7fb1901457f0 legacy=0x7fb190137280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb18d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:01:23.423 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb18d7fa640 1 monclient: found mon.c 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb18d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb18d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190142ce0 con 0x7fb19013f650 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb18d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb18d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb18d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb196c15640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb196c15640 10 monclient: _renew_subs 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb196c15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1900ab600 con 0x7fb19013f650 2024-03-29T11:01:23.424 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb196c15640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb196c15640 10 monclient: _renew_subs 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb196c15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19013ff80 con 0x7fb19013f650 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.435+0000 7fb196c15640 1 librados: init done 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.436+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1840047e0 con 0x7fb19013f650 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.436+0000 7fb18d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.436+0000 7fb18effd640 4 set_mon_vals no callback set 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.436+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb184004af0 con 0x7fb19013f650 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.436+0000 7fb18d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.436+0000 7fb18d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.436+0000 7fb18d7fa640 10 monclient: dump: 2024-03-29T11:01:23.425 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.436+0000 7fb196c15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.437+0000 7fb196c15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.437+0000 7fb196c15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:23.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.437+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb170005080 con 0x7fb19013f650 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.438+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb18401aac0 con 0x7fb19013f650 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.438+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fb1840848a0 con 0x7fb19013f650 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.438+0000 7fb18d7fa640 10 client.4388.objecter ms_dispatch 0x7fb190000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.438+0000 7fb18d7fa640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:23.427 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.438+0000 7fb18d7fa640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.438+0000 7fb1949b1640 10 monclient: get_auth_request con 0x7fb17406dc40 auth_method 0 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.438+0000 7fb18d7fa640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.438+0000 7fb18d7fa640 10 client.4388.objecter ms_handle_connect 0x7fb17406dc40 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.446+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1840501e0 con 0x7fb19013f650 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.446+0000 7fb18d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:23.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.446+0000 7fb18d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:23.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.765+0000 7fb196c15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.765+0000 7fb196c15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.765+0000 7fb196c15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:23.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.765+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb170005330 con 0x7fb19013f650 2024-03-29T11:01:23.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.767+0000 7fb18d7fa640 1 -- 172.21.15.107:0/2156968134 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fb184004950 con 0x7fb19013f650 2024-03-29T11:01:23.747 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:01:23.747 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:23.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.767+0000 7fb18d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.767+0000 7fb18d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:23.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 10 librados: watch_flush enter 2024-03-29T11:01:23.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 10 librados: watch_flush exit 2024-03-29T11:01:23.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T11:01:23.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 10 client.4388.objecter successfully canceled tick 2024-03-29T11:01:23.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 >> v1:172.21.15.107:6816/882194755 conn(0x7fb17406dc40 legacy=0x7fb174070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 10 monclient: shutdown 2024-03-29T11:01:23.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:23.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 >> v1:172.21.15.107:6790/0 conn(0x7fb19013f650 legacy=0x7fb190136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 shutdown_connections 2024-03-29T11:01:23.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 >> 172.21.15.107:0/2156968134 conn(0x7fb190066a00 msgr2=0x7fb19013aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:23.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 shutdown_connections 2024-03-29T11:01:23.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.770+0000 7fb196c15640 1 -- 172.21.15.107:0/2156968134 wait complete. 2024-03-29T11:01:23.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:23.771+0000 7fb196c15640 1 librados: shutdown 2024-03-29T11:01:23.768 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:30.575 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:30.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.747+0000 7fb1edcf1640 10 monclient: get_monmap_and_config 2024-03-29T11:01:30.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.747+0000 7fb1edcf1640 10 monclient: build_initial_monmap 2024-03-29T11:01:30.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.747+0000 7fb1edcf1640 10 monclient: monmap: 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:01:30.748366+0000 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:01:30.748366+0000 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:01:30.728 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:30.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.748+0000 7fb1edcf1640 1 Processor -- start 2024-03-29T11:01:30.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.749+0000 7fb1edcf1640 1 -- start start 2024-03-29T11:01:30.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.749+0000 7fb1edcf1640 10 monclient: init 2024-03-29T11:01:30.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.749+0000 7fb1edcf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.749+0000 7fb1edcf1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1edcf1640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e80b4330 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1edcf1640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e81470b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1edcf1640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e81439b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1edcf1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e806f2a0 con 0x7fb1e80b4330 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1edcf1640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e8146b90 con 0x7fb1e81470b0 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1edcf1640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80b2ca0 con 0x7fb1e81439b0 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1edcf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1edcf1640 10 monclient(hunting): authenticate will time out at 1022.018082s 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1e6ffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb1e81470b0 0x7fb1e80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:38128/0 (socket says 172.21.15.107:38128) 2024-03-29T11:01:30.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1e6ffd640 1 -- 172.21.15.107:0/2239119417 learned_addr learned my addr 172.21.15.107:0/2239119417 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1e67fc640 1 -- 172.21.15.107:0/2239119417 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009078227 0 0) 0x7fb1e806f2a0 con 0x7fb1e80b4330 2024-03-29T11:01:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.732 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1e67fc640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:01:30.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.750+0000 7fb1e67fc640 1 -- 172.21.15.107:0/2239119417 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d0003520 con 0x7fb1e80b4330 2024-03-29T11:01:30.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 1 -- 172.21.15.107:0/2239119417 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971515565 0 0) 0x7fb1d0003520 con 0x7fb1e80b4330 2024-03-29T11:01:30.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 1 -- 172.21.15.107:0/2239119417 >> v1:172.21.15.107:6790/0 conn(0x7fb1e81439b0 legacy=0x7fb1e8151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:30.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 1 -- 172.21.15.107:0/2239119417 >> v1:172.21.15.129:6789/0 conn(0x7fb1e81470b0 legacy=0x7fb1e80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:30.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 1 monclient: found mon.noname-a 2024-03-29T11:01:30.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:30.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:01:30.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 1 -- 172.21.15.107:0/2239119417 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e806e0c0 con 0x7fb1e80b4330 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 1 -- 172.21.15.107:0/2239119417 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb1d8001690 con 0x7fb1e80b4330 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 10 monclient: dump: 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.734 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:30.735 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:30.735 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:30.735 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:30.735 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:30.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1edcf1640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:01:30.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1edcf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:30.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 1 -- 172.21.15.107:0/2239119417 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1d8001860 con 0x7fb1e80b4330 2024-03-29T11:01:30.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:30.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1edcf1640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.751+0000 7fb1edcf1640 4 set_mon_vals no callback set 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1e67fc640 1 -- 172.21.15.107:0/2239119417 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb1d8001c70 con 0x7fb1e80b4330 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1e67fc640 10 monclient: dump: 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:30.736 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:30.737 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:30.737 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:30.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1edcf1640 10 monclient: shutdown 2024-03-29T11:01:30.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1edcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:30.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2239119417 >> v1:172.21.15.107:6789/0 conn(0x7fb1e80b4330 legacy=0x7fb1e80b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2239119417 shutdown_connections 2024-03-29T11:01:30.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2239119417 >> 172.21.15.107:0/2239119417 conn(0x7fb1e8066a00 msgr2=0x7fb1e8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:30.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2239119417 shutdown_connections 2024-03-29T11:01:30.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.752+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2239119417 wait complete. 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.753+0000 7fb1edcf1640 10 monclient: build_initial_monmap 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.753+0000 7fb1edcf1640 10 monclient: monmap: 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:01:30.747 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.754+0000 7fb1edcf1640 1 librados: starting msgr at 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.754+0000 7fb1edcf1640 1 librados: starting objecter 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.754+0000 7fb1edcf1640 1 Processor -- start 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.754+0000 7fb1edcf1640 1 -- start start 2024-03-29T11:01:30.748 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.754+0000 7fb1edcf1640 1 librados: setting wanted keys 2024-03-29T11:01:30.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.754+0000 7fb1edcf1640 1 librados: calling monclient init 2024-03-29T11:01:30.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.754+0000 7fb1edcf1640 10 monclient: init 2024-03-29T11:01:30.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1edcf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:30.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1edcf1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:30.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1edcf1640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e80b4330 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:30.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1edcf1640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e81439b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:30.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1edcf1640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e81470b0 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:30.749 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1edcf1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e8146380 con 0x7fb1e81470b0 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1e7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb1e81470b0 0x7fb1e81e5fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57968/0 (socket says 172.21.15.107:57968) 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1e7fff640 1 -- 172.21.15.107:0/2585100917 learned_addr learned my addr 172.21.15.107:0/2585100917 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1cffff640 10 client.?.objecter ms_handle_connect 0x7fb1e81470b0 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1edcf1640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e81216a0 con 0x7fb1e81439b0 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e8145e10 con 0x7fb1e80b4330 2024-03-29T11:01:30.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.755+0000 7fb1edcf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1edcf1640 10 monclient(hunting): authenticate will time out at 1022.024003s 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 client.?.objecter ms_handle_connect 0x7fb1e81439b0 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 client.?.objecter ms_handle_connect 0x7fb1e80b4330 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119898213 0 0) 0x7fb1e8146380 con 0x7fb1e81470b0 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:01:30.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d4004eb0 con 0x7fb1e81470b0 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820472861 0 0) 0x7fb1e8145e10 con 0x7fb1e80b4330 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e8146380 con 0x7fb1e80b4330 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947749300 0 0) 0x7fb1e81216a0 con 0x7fb1e81439b0 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.752 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:01:30.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.756+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e8145e10 con 0x7fb1e81439b0 2024-03-29T11:01:30.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3396915557 0 0) 0x7fb1d4004eb0 con 0x7fb1e81470b0 2024-03-29T11:01:30.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e81216a0 con 0x7fb1e81470b0 2024-03-29T11:01:30.753 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb1dc003f60 con 0x7fb1e81470b0 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): dump: 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:30.754 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2955732598 0 0) 0x7fb1e8145e10 con 0x7fb1e81439b0 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1d4004eb0 con 0x7fb1e81439b0 2024-03-29T11:01:30.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb1e0001a80 con 0x7fb1e81439b0 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): dump: 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:30.756 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 21001250 0 0) 0x7fb1e8146380 con 0x7fb1e80b4330 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e8145e10 con 0x7fb1e80b4330 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb1d8001c70 con 0x7fb1e80b4330 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient(hunting): dump: 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:30.757 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2015895518 0 0) 0x7fb1e81216a0 con 0x7fb1e81470b0 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 >> v1:172.21.15.107:6790/0 conn(0x7fb1e80b4330 legacy=0x7fb1e81dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 >> v1:172.21.15.129:6789/0 conn(0x7fb1e81439b0 legacy=0x7fb1e81df340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 monclient: found mon.a 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8145fa0 con 0x7fb1e81470b0 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient: _finish_auth 0 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1edcf1640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1edcf1640 10 monclient: _renew_subs 2024-03-29T11:01:30.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1edcf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:01:30.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e80b6a40 con 0x7fb1e81470b0 2024-03-29T11:01:30.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1edcf1640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:30.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1edcf1640 10 monclient: _renew_subs 2024-03-29T11:01:30.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1edcf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:01:30.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e8141d30 con 0x7fb1e81470b0 2024-03-29T11:01:30.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1dc0040c0 con 0x7fb1e81470b0 2024-03-29T11:01:30.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:30.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb1dc001070 con 0x7fb1e81470b0 2024-03-29T11:01:30.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1cffff640 10 monclient: dump: 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:30.762 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.757+0000 7fb1edcf1640 1 librados: init done 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.758+0000 7fb1e5ffb640 4 set_mon_vals no callback set 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.759+0000 7fb1edcf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.759+0000 7fb1edcf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.759+0000 7fb1edcf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.759+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e8075f10 con 0x7fb1e81470b0 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.760+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb1dc0011d0 con 0x7fb1e81470b0 2024-03-29T11:01:30.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.761+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fb1dc084850 con 0x7fb1e81470b0 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.761+0000 7fb1cffff640 10 client.4371.objecter ms_dispatch 0x7fb1e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.761+0000 7fb1cffff640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.761+0000 7fb1cffff640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.761+0000 7fb1cffff640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.761+0000 7fb1e77fe640 10 monclient: get_auth_request con 0x7fb1d406e140 auth_method 0 2024-03-29T11:01:30.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.762+0000 7fb1cffff640 10 client.4371.objecter ms_handle_connect 0x7fb1d406e140 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.770+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1dc050130 con 0x7fb1e81470b0 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.770+0000 7fb1cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.765 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:30.770+0000 7fb1cffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:31.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.128+0000 7fb1edcf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:31.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.128+0000 7fb1edcf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:31.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.128+0000 7fb1edcf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:01:31.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.128+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e81de990 con 0x7fb1e81470b0 2024-03-29T11:01:31.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.130+0000 7fb1cffff640 1 -- 172.21.15.107:0/2585100917 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fb1dc052060 con 0x7fb1e81470b0 2024-03-29T11:01:31.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.130+0000 7fb1cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:31.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.130+0000 7fb1cffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:31.110 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:01:31.110 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.132+0000 7fb1edcf1640 10 librados: watch_flush enter 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.132+0000 7fb1edcf1640 10 librados: watch_flush exit 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.132+0000 7fb1edcf1640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.132+0000 7fb1edcf1640 10 client.4371.objecter successfully canceled tick 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.132+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 >> v1:172.21.15.107:6816/882194755 conn(0x7fb1d406e140 legacy=0x7fb1d4070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.132+0000 7fb1edcf1640 10 monclient: shutdown 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.132+0000 7fb1edcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.132+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 >> v1:172.21.15.107:6789/0 conn(0x7fb1e81470b0 legacy=0x7fb1e81e5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.133+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 shutdown_connections 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.133+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 >> 172.21.15.107:0/2585100917 conn(0x7fb1e8066a00 msgr2=0x7fb1e8070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:31.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.133+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 shutdown_connections 2024-03-29T11:01:31.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.133+0000 7fb1edcf1640 1 -- 172.21.15.107:0/2585100917 wait complete. 2024-03-29T11:01:31.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:31.133+0000 7fb1edcf1640 1 librados: shutdown 2024-03-29T11:01:31.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:37.933 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:38.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.109+0000 7fcd0f463640 10 monclient: get_monmap_and_config 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.109+0000 7fcd0f463640 10 monclient: build_initial_monmap 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.109+0000 7fcd0f463640 10 monclient: monmap: 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:01:38.110781+0000 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:01:38.110781+0000 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:01:38.092 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.111+0000 7fcd0f463640 1 Processor -- start 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.112+0000 7fcd0f463640 1 -- start start 2024-03-29T11:01:38.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.112+0000 7fcd0f463640 10 monclient: init 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 10 monclient(hunting): picked mon.noname-b con 0x7fcd080af360 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 10 monclient(hunting): picked mon.noname-a con 0x7fcd0813ee80 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 10 monclient(hunting): picked mon.noname-c con 0x7fcd0813f2c0 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd080ab9c0 con 0x7fcd0813ee80 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd080abca0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd080abf80 con 0x7fcd080af360 2024-03-29T11:01:38.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0f463640 10 monclient(hunting): authenticate will time out at 1029.381335s 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0d1ff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcd080af360 0x7fcd0813c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47168/0 (socket says 172.21.15.107:47168) 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.113+0000 7fcd0d1ff640 1 -- 172.21.15.107:0/3463626174 learned_addr learned my addr 172.21.15.107:0/3463626174 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683646280 0 0) 0x7fcd080ab9c0 con 0x7fcd0813ee80 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccf4003520 con 0x7fcd0813ee80 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847652692 0 0) 0x7fcd080abf80 con 0x7fcd080af360 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd080ab9c0 con 0x7fcd080af360 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2925466855 0 0) 0x7fccf4003520 con 0x7fcd0813ee80 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 >> v1:172.21.15.107:6790/0 conn(0x7fcd080af360 legacy=0x7fcd0813c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.114+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 >> v1:172.21.15.129:6789/0 conn(0x7fcd0813f2c0 legacy=0x7fcd080ae6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:38.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 1 monclient: found mon.noname-a 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0813c9c0 con 0x7fcd0813ee80 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 10 monclient: _finish_auth 0 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcd00001a10 con 0x7fcd0813ee80 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fccfffff640 10 monclient: dump: 2024-03-29T11:01:38.098 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fcd0f463640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.117+0000 7fcd0f463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.118+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd00001b70 con 0x7fcd0813ee80 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.118+0000 7fccfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:38.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.118+0000 7fccfffff640 1 -- 172.21.15.107:0/3463626174 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcd00001f80 con 0x7fcd0813ee80 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.118+0000 7fcd0f463640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.118+0000 7fccfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.118+0000 7fcd0f463640 4 set_mon_vals no callback set 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.118+0000 7fccfffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.118+0000 7fccfffff640 10 monclient: dump: 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:38.100 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.119+0000 7fcd0f463640 10 monclient: shutdown 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.119+0000 7fcd0f463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.119+0000 7fcd0f463640 1 -- 172.21.15.107:0/3463626174 >> v1:172.21.15.107:6789/0 conn(0x7fcd0813ee80 legacy=0x7fcd080adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.119+0000 7fcd0f463640 1 -- 172.21.15.107:0/3463626174 shutdown_connections 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.119+0000 7fcd0f463640 1 -- 172.21.15.107:0/3463626174 >> 172.21.15.107:0/3463626174 conn(0x7fcd080b35e0 msgr2=0x7fcd080b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.120+0000 7fcd0f463640 1 -- 172.21.15.107:0/3463626174 shutdown_connections 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.120+0000 7fcd0f463640 1 -- 172.21.15.107:0/3463626174 wait complete. 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.121+0000 7fcd0f463640 10 monclient: build_initial_monmap 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.121+0000 7fcd0f463640 10 monclient: monmap: 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:01:38.102 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.121+0000 7fcd0f463640 1 librados: starting msgr at 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.121+0000 7fcd0f463640 1 librados: starting objecter 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.122+0000 7fcd0f463640 1 Processor -- start 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.122+0000 7fcd0f463640 1 -- start start 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.122+0000 7fcd0f463640 1 librados: setting wanted keys 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.122+0000 7fcd0f463640 1 librados: calling monclient init 2024-03-29T11:01:38.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.122+0000 7fcd0f463640 10 monclient: init 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.122+0000 7fcd0f463640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.122+0000 7fcd0f463640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.122+0000 7fcd0f463640 10 monclient(hunting): picked mon.noname-b con 0x7fcd080af360 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.122+0000 7fcd0f463640 10 monclient(hunting): picked mon.noname-a con 0x7fcd0813ee80 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.123+0000 7fcd0f463640 10 monclient(hunting): picked mon.noname-c con 0x7fcd0813f2c0 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.123+0000 7fcd0f463640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd081390d0 con 0x7fcd0813ee80 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.123+0000 7fcd0f463640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd08137020 con 0x7fcd080af360 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.123+0000 7fcd0f463640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd080b11d0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.123+0000 7fcd0f463640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:38.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.123+0000 7fcd0f463640 10 monclient(hunting): authenticate will time out at 1029.391019s 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.125+0000 7fcd0da00640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcd0813f2c0 0x7fcd081de4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47776/0 (socket says 172.21.15.107:47776) 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.125+0000 7fcd0da00640 1 -- 172.21.15.107:0/313530328 learned_addr learned my addr 172.21.15.107:0/313530328 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.125+0000 7fccfdffb640 10 client.?.objecter ms_handle_connect 0x7fcd0813f2c0 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.125+0000 7fccfdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831381618 0 0) 0x7fcd080b11d0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:01:38.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccec004eb0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 10 client.?.objecter ms_handle_connect 0x7fcd0813ee80 2024-03-29T11:01:38.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:38.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 10 client.?.objecter ms_handle_connect 0x7fcd080af360 2024-03-29T11:01:38.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:38.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2075822937 0 0) 0x7fccec004eb0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.126+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd080b11d0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fccf8001750 con 0x7fcd0813f2c0 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): dump: 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:38.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127251588 0 0) 0x7fcd08137020 con 0x7fcd080af360 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccec004eb0 con 0x7fcd080af360 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306229222 0 0) 0x7fcd081390d0 con 0x7fcd0813ee80 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd08137020 con 0x7fcd0813ee80 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1783338297 0 0) 0x7fcd080b11d0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 >> v1:172.21.15.129:6789/0 conn(0x7fcd080af360 legacy=0x7fcd08137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 >> v1:172.21.15.107:6789/0 conn(0x7fcd0813ee80 legacy=0x7fcd08131870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 monclient: found mon.c 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd080b1360 con 0x7fcd0813f2c0 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fccfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fcd0f463640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fcd0f463640 10 monclient: _renew_subs 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fcd0f463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.127+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd081d6dc0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.128+0000 7fcd0f463640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:38.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.128+0000 7fcd0f463640 10 monclient: _renew_subs 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.128+0000 7fcd0f463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.128+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd08137e30 con 0x7fcd0813f2c0 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.128+0000 7fcd0f463640 1 librados: init done 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.128+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccf8001d60 con 0x7fcd0813f2c0 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.128+0000 7fccfdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.128+0000 7fccfeffd640 4 set_mon_vals no callback set 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.129+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fccf8001930 con 0x7fcd0813f2c0 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.129+0000 7fccfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.129+0000 7fccfdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.129+0000 7fccfdffb640 10 monclient: dump: 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:38.115 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.131+0000 7fcd0f463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.131+0000 7fcd0f463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:38.116 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.131+0000 7fcd0f463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:38.117 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.131+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce8005080 con 0x7fcd0813f2c0 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.131+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fccf8004860 con 0x7fcd0813f2c0 2024-03-29T11:01:38.118 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.132+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fccf80847a0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.132+0000 7fccfdffb640 10 client.4397.objecter ms_dispatch 0x7fcd08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.132+0000 7fccfdffb640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.132+0000 7fccfdffb640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.132+0000 7fccfdffb640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.132+0000 7fcd0d1ff640 10 monclient: get_auth_request con 0x7fccec06dcc0 auth_method 0 2024-03-29T11:01:38.119 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.133+0000 7fccfdffb640 10 client.4397.objecter ms_handle_connect 0x7fccec06dcc0 2024-03-29T11:01:38.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.140+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccf8050080 con 0x7fcd0813f2c0 2024-03-29T11:01:38.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.141+0000 7fccfdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:38.120 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.141+0000 7fccfdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:38.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.509+0000 7fcd0f463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:38.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.509+0000 7fcd0f463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:38.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.509+0000 7fcd0f463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:38.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.509+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcce8005330 con 0x7fcd0813f2c0 2024-03-29T11:01:38.491 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:01:38.491 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:38.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.511+0000 7fccfdffb640 1 -- 172.21.15.107:0/313530328 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fccf8051fb0 con 0x7fcd0813f2c0 2024-03-29T11:01:38.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.511+0000 7fccfdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:38.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.511+0000 7fccfdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:38.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.514+0000 7fcd0f463640 10 librados: watch_flush enter 2024-03-29T11:01:38.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.514+0000 7fcd0f463640 10 librados: watch_flush exit 2024-03-29T11:01:38.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.514+0000 7fcd0f463640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T11:01:38.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.514+0000 7fcd0f463640 10 client.4397.objecter successfully canceled tick 2024-03-29T11:01:38.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.515+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 >> v1:172.21.15.107:6816/882194755 conn(0x7fccec06dcc0 legacy=0x7fccec070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.515+0000 7fcd0f463640 10 monclient: shutdown 2024-03-29T11:01:38.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.515+0000 7fcd0f463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:38.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.515+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 >> v1:172.21.15.107:6790/0 conn(0x7fcd0813f2c0 legacy=0x7fcd081de4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:38.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.515+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 shutdown_connections 2024-03-29T11:01:38.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.515+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 >> 172.21.15.107:0/313530328 conn(0x7fcd080b35e0 msgr2=0x7fcd080b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:38.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.515+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 shutdown_connections 2024-03-29T11:01:38.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.516+0000 7fcd0f463640 1 -- 172.21.15.107:0/313530328 wait complete. 2024-03-29T11:01:38.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:38.516+0000 7fcd0f463640 1 librados: shutdown 2024-03-29T11:01:38.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:45.318 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:45.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.479+0000 7f43b208d640 10 monclient: get_monmap_and_config 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.479+0000 7f43b208d640 10 monclient: build_initial_monmap 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.479+0000 7f43b208d640 10 monclient: monmap: 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:01:45.480839+0000 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:01:45.480839+0000 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:01:45.460 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:01:45.461 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:45.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.480+0000 7f43b208d640 1 Processor -- start 2024-03-29T11:01:45.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.481+0000 7f43b208d640 1 -- start start 2024-03-29T11:01:45.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.481+0000 7f43b208d640 10 monclient: init 2024-03-29T11:01:45.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:45.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:45.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 10 monclient(hunting): picked mon.noname-c con 0x7f43ac1422b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:45.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 10 monclient(hunting): picked mon.noname-a con 0x7f43ac142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:45.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 10 monclient(hunting): picked mon.noname-b con 0x7f43ac0b17d0 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:45.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac142c80 con 0x7f43ac142690 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac0aae40 con 0x7f43ac1422b0 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac13fbd0 con 0x7f43ac0b17d0 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43b208d640 10 monclient(hunting): authenticate will time out at 1036.750393s 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43aaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f43ac142690 0x7f43ac0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33098/0 (socket says 172.21.15.107:33098) 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43abfff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f43ac0b17d0 0x7f43ac0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42748/0 (socket says 172.21.15.107:42748) 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.482+0000 7f43abfff640 1 -- 172.21.15.107:0/4277901413 learned_addr learned my addr 172.21.15.107:0/4277901413 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166852092 0 0) 0x7f43ac13fbd0 con 0x7f43ac0b17d0 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:01:45.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4394003520 con 0x7f43ac0b17d0 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147089286 0 0) 0x7f43ac142c80 con 0x7f43ac142690 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.483+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ac13fbd0 con 0x7f43ac142690 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2819071610 0 0) 0x7f43ac13fbd0 con 0x7f43ac142690 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 >> v1:172.21.15.107:6790/0 conn(0x7f43ac0b17d0 legacy=0x7f43ac0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 >> v1:172.21.15.129:6789/0 conn(0x7f43ac1422b0 legacy=0x7f43ac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 1 monclient: found mon.noname-a 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac151940 con 0x7f43ac142690 2024-03-29T11:01:45.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43b208d640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43b208d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f43a00015f0 con 0x7f43ac142690 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: dump: 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.470 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43b208d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43a00017d0 con 0x7f43ac142690 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43b208d640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43b208d640 4 set_mon_vals no callback set 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 1 -- 172.21.15.107:0/4277901413 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f43a00015f0 con 0x7f43ac142690 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:45.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.484+0000 7f43aa7fc640 10 monclient: dump: 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.485+0000 7f43b208d640 10 monclient: shutdown 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.485+0000 7f43b208d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.485+0000 7f43b208d640 1 -- 172.21.15.107:0/4277901413 >> v1:172.21.15.107:6789/0 conn(0x7f43ac142690 legacy=0x7f43ac0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.485+0000 7f43b208d640 1 -- 172.21.15.107:0/4277901413 shutdown_connections 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.485+0000 7f43b208d640 1 -- 172.21.15.107:0/4277901413 >> 172.21.15.107:0/4277901413 conn(0x7f43ac066a00 msgr2=0x7f43ac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:45.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.485+0000 7f43b208d640 1 -- 172.21.15.107:0/4277901413 shutdown_connections 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.485+0000 7f43b208d640 1 -- 172.21.15.107:0/4277901413 wait complete. 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.486+0000 7f43b208d640 10 monclient: build_initial_monmap 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.486+0000 7f43b208d640 10 monclient: monmap: 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:01:45.473 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:45.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.486+0000 7f43b208d640 1 librados: starting msgr at 2024-03-29T11:01:45.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.486+0000 7f43b208d640 1 librados: starting objecter 2024-03-29T11:01:45.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.486+0000 7f43b208d640 1 Processor -- start 2024-03-29T11:01:45.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.487+0000 7f43b208d640 1 -- start start 2024-03-29T11:01:45.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.487+0000 7f43b208d640 1 librados: setting wanted keys 2024-03-29T11:01:45.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.487+0000 7f43b208d640 1 librados: calling monclient init 2024-03-29T11:01:45.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.487+0000 7f43b208d640 10 monclient: init 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 10 monclient(hunting): picked mon.noname-a con 0x7f43ac0b17d0 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 10 monclient(hunting): picked mon.noname-c con 0x7f43ac1422b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 10 monclient(hunting): picked mon.noname-b con 0x7f43ac142690 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac0b6690 con 0x7f43ac0b17d0 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac0b6970 con 0x7f43ac142690 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac0b6c50 con 0x7f43ac1422b0 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43b208d640 10 monclient(hunting): authenticate will time out at 1036.756228s 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43ab7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f43ac0b17d0 0x7f43ac150270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37072/0 (socket says 172.21.15.107:37072) 2024-03-29T11:01:45.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43aaffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f43ac1422b0 0x7f43ac12db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47662/0 (socket says 172.21.15.107:47662) 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f43ab7fe640 1 -- 172.21.15.107:0/2575296002 learned_addr learned my addr 172.21.15.107:0/2575296002 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f4393fff640 10 client.?.objecter ms_handle_connect 0x7f43ac1422b0 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f4393fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f4393fff640 10 client.?.objecter ms_handle_connect 0x7f43ac0b17d0 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.488+0000 7f4393fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268869322 0 0) 0x7f43ac0b6c50 con 0x7f43ac1422b0 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4398004eb0 con 0x7f43ac1422b0 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 client.?.objecter ms_handle_connect 0x7f43ac142690 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:45.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423909000 0 0) 0x7f43ac0b6690 con 0x7f43ac0b17d0 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ac0b6c50 con 0x7f43ac0b17d0 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3484189749 0 0) 0x7f4398004eb0 con 0x7f43ac1422b0 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43ac0b6690 con 0x7f43ac1422b0 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f43a0001ba0 con 0x7f43ac1422b0 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): dump: 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:45.485 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2415628857 0 0) 0x7f43ac0b6c50 con 0x7f43ac0b17d0 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4398005970 con 0x7f43ac0b17d0 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f439c001970 con 0x7f43ac0b17d0 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): dump: 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761965116 0 0) 0x7f43ac0b6970 con 0x7f43ac142690 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:45.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.489+0000 7f4393fff640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ac0b6c50 con 0x7f43ac142690 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3257547202 0 0) 0x7f43ac0b6690 con 0x7f43ac1422b0 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 >> v1:172.21.15.129:6789/0 conn(0x7f43ac142690 legacy=0x7f43ac1e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 >> v1:172.21.15.107:6789/0 conn(0x7f43ac0b17d0 legacy=0x7f43ac150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 1 monclient: found mon.c 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac0b5960 con 0x7f43ac1422b0 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f4393fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f43b208d640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:01:45.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f43b208d640 10 monclient: _renew_subs 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f43b208d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ac150a50 con 0x7f43ac1422b0 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f43b208d640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f43b208d640 10 monclient: _renew_subs 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f43b208d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ac1511c0 con 0x7f43ac1422b0 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.490+0000 7f43b208d640 1 librados: init done 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.491+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43a0004a90 con 0x7f43ac1422b0 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.491+0000 7f4393fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.491+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f43a0001ba0 con 0x7f43ac1422b0 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.491+0000 7f43a9ffb640 4 set_mon_vals no callback set 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.491+0000 7f4393fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.491+0000 7f4393fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.491+0000 7f4393fff640 10 monclient: dump: 2024-03-29T11:01:45.489 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.491+0000 7f43b208d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.492+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f43a001aa70 con 0x7f43ac1422b0 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.493+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f43a0084550 con 0x7f43ac1422b0 2024-03-29T11:01:45.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.493+0000 7f4393fff640 10 client.4403.objecter ms_dispatch 0x7f43ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.493+0000 7f4393fff640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.493+0000 7f4393fff640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.492+0000 7f43b208d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.493+0000 7f43b208d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.493+0000 7f4393fff640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.493+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438c005080 con 0x7f43ac1422b0 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.493+0000 7f43ab7fe640 10 monclient: get_auth_request con 0x7f439806e500 auth_method 0 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.493+0000 7f4393fff640 10 client.4403.objecter ms_handle_connect 0x7f439806e500 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.501+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43a004ff70 con 0x7f43ac1422b0 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.501+0000 7f4393fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:45.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.501+0000 7f4393fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:45.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.787+0000 7f43b208d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.787+0000 7f43b208d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.787+0000 7f43b208d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:45.767 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.787+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f438c005330 con 0x7f43ac1422b0 2024-03-29T11:01:45.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.788+0000 7f4393fff640 1 -- 172.21.15.107:0/2575296002 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f43a004ff70 con 0x7f43ac1422b0 2024-03-29T11:01:45.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.788+0000 7f4393fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.768 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.788+0000 7f4393fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:45.768 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:01:45.768 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:45.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 10 librados: watch_flush enter 2024-03-29T11:01:45.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 10 librados: watch_flush exit 2024-03-29T11:01:45.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T11:01:45.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 10 client.4403.objecter successfully canceled tick 2024-03-29T11:01:45.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 >> v1:172.21.15.107:6816/882194755 conn(0x7f439806e500 legacy=0x7f43980709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 10 monclient: shutdown 2024-03-29T11:01:45.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:45.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 >> v1:172.21.15.107:6790/0 conn(0x7f43ac1422b0 legacy=0x7f43ac12db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 shutdown_connections 2024-03-29T11:01:45.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.790+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 >> 172.21.15.107:0/2575296002 conn(0x7f43ac066a00 msgr2=0x7f43ac0b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:45.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.791+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 shutdown_connections 2024-03-29T11:01:45.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.791+0000 7f43b208d640 1 -- 172.21.15.107:0/2575296002 wait complete. 2024-03-29T11:01:45.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:45.791+0000 7f43b208d640 1 librados: shutdown 2024-03-29T11:01:45.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:52.590 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:52.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.734+0000 7fcae419d640 10 monclient: get_monmap_and_config 2024-03-29T11:01:52.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.734+0000 7fcae419d640 10 monclient: build_initial_monmap 2024-03-29T11:01:52.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.734+0000 7fcae419d640 10 monclient: monmap: 2024-03-29T11:01:52.714 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:52.714 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:52.714 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:01:52.735508+0000 2024-03-29T11:01:52.715 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:01:52.735508+0000 2024-03-29T11:01:52.715 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:52.715 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:52.715 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:52.715 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:01:52.715 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:01:52.715 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.735+0000 7fcae419d640 1 Processor -- start 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.735+0000 7fcae419d640 1 -- start start 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.735+0000 7fcae419d640 10 monclient: init 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 10 monclient(hunting): picked mon.noname-a con 0x7fcadc1457f0 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 10 monclient(hunting): picked mon.noname-b con 0x7fcadc13f650 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 10 monclient(hunting): picked mon.noname-c con 0x7fcadc0a9d70 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcadc0aaef0 con 0x7fcadc1457f0 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcadc145bd0 con 0x7fcadc0a9d70 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcadc145eb0 con 0x7fcadc13f650 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:52.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae419d640 10 monclient(hunting): authenticate will time out at 1044.004129s 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae1738640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcadc13f650 0x7fcadc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39448/0 (socket says 172.21.15.107:39448) 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae1738640 1 -- 172.21.15.107:0/364218670 learned_addr learned my addr 172.21.15.107:0/364218670 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae0f37640 1 -- 172.21.15.107:0/364218670 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709593887 0 0) 0x7fcadc145eb0 con 0x7fcadc13f650 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae0f37640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae0f37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae0f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae0f37640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.736+0000 7fcae0f37640 1 -- 172.21.15.107:0/364218670 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac8003520 con 0x7fcadc13f650 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.737+0000 7fcae0f37640 1 -- 172.21.15.107:0/364218670 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 555703311 0 0) 0x7fcac8003520 con 0x7fcadc13f650 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.737+0000 7fcae0f37640 1 -- 172.21.15.107:0/364218670 >> v1:172.21.15.129:6789/0 conn(0x7fcadc0a9d70 legacy=0x7fcadc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.737+0000 7fcae0f37640 1 -- 172.21.15.107:0/364218670 >> v1:172.21.15.107:6789/0 conn(0x7fcadc1457f0 legacy=0x7fcadc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:52.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: _finish_hunting 0 2024-03-29T11:01:52.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 1 monclient: found mon.noname-b 2024-03-29T11:01:52.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:52.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:01:52.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 1 -- 172.21.15.107:0/364218670 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcadc146340 con 0x7fcadc13f650 2024-03-29T11:01:52.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: _finish_auth 0 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: _check_auth_tickets 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 1 -- 172.21.15.107:0/364218670 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcacc001690 con 0x7fcadc13f650 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: dump: 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:52.720 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae419d640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae419d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 1 -- 172.21.15.107:0/364218670 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcacc001860 con 0x7fcadc13f650 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 1 -- 172.21.15.107:0/364218670 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcacc001cb0 con 0x7fcadc13f650 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae0f37640 10 monclient: dump: 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:52.721 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.722 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:52.722 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:52.722 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:52.722 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:52.722 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:52.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae419d640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:52.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.739+0000 7fcae419d640 4 set_mon_vals no callback set 2024-03-29T11:01:52.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.740+0000 7fcae419d640 10 monclient: shutdown 2024-03-29T11:01:52.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.740+0000 7fcae419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:52.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.740+0000 7fcae419d640 1 -- 172.21.15.107:0/364218670 >> v1:172.21.15.107:6790/0 conn(0x7fcadc13f650 legacy=0x7fcadc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.740+0000 7fcae419d640 1 -- 172.21.15.107:0/364218670 shutdown_connections 2024-03-29T11:01:52.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.740+0000 7fcae419d640 1 -- 172.21.15.107:0/364218670 >> 172.21.15.107:0/364218670 conn(0x7fcadc066a00 msgr2=0x7fcadc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:52.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.740+0000 7fcae419d640 1 -- 172.21.15.107:0/364218670 shutdown_connections 2024-03-29T11:01:52.726 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.740+0000 7fcae419d640 1 -- 172.21.15.107:0/364218670 wait complete. 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.741+0000 7fcae419d640 10 monclient: build_initial_monmap 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.741+0000 7fcae419d640 10 monclient: monmap: 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.742+0000 7fcae419d640 1 librados: starting msgr at 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.742+0000 7fcae419d640 1 librados: starting objecter 2024-03-29T11:01:52.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.742+0000 7fcae419d640 1 Processor -- start 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.742+0000 7fcae419d640 1 -- start start 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.742+0000 7fcae419d640 1 librados: setting wanted keys 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.742+0000 7fcae419d640 1 librados: calling monclient init 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.742+0000 7fcae419d640 10 monclient: init 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 10 monclient(hunting): picked mon.noname-c con 0x7fcadc0a9d70 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 10 monclient(hunting): picked mon.noname-b con 0x7fcadc13f650 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 10 monclient(hunting): picked mon.noname-a con 0x7fcadc1457f0 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcadc142410 con 0x7fcadc1457f0 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcadc142ca0 con 0x7fcadc13f650 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcadc142830 con 0x7fcadc0a9d70 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:52.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae419d640 10 monclient(hunting): authenticate will time out at 1044.011615s 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae1f39640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fcadc0a9d70 0x7fcadc135060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39676/0 (socket says 172.21.15.107:39676) 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcae1f39640 1 -- 172.21.15.107:0/4259718165 learned_addr learned my addr 172.21.15.107:0/4259718165 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcad27fc640 10 client.?.objecter ms_handle_connect 0x7fcadc0a9d70 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.743+0000 7fcad27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350537195 0 0) 0x7fcadc142830 con 0x7fcadc0a9d70 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac0004eb0 con 0x7fcadc0a9d70 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1984179974 0 0) 0x7fcac0004eb0 con 0x7fcadc0a9d70 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcadc142830 con 0x7fcadc0a9d70 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcad8001b50 con 0x7fcadc0a9d70 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 10 monclient(hunting): dump: 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.744+0000 7fcad27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.745+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3827469882 0 0) 0x7fcadc142830 con 0x7fcadc0a9d70 2024-03-29T11:01:52.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.745+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 >> v1:172.21.15.129:6789/0 conn(0x7fcadc13f650 legacy=0x7fcadc1dadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.745+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 >> v1:172.21.15.107:6789/0 conn(0x7fcadc1457f0 legacy=0x7fcadc1db4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 1 monclient: found mon.c 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcadc1429c0 con 0x7fcadc0a9d70 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcae419d640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcae419d640 10 monclient: _renew_subs 2024-03-29T11:01:52.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcae419d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcadc140010 con 0x7fcadc0a9d70 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcae419d640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcae419d640 10 monclient: _renew_subs 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcae419d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcadc135770 con 0x7fcadc0a9d70 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcae419d640 1 librados: init done 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcad8001db0 con 0x7fcadc0a9d70 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcad8004dc0 con 0x7fcadc0a9d70 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:52.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad27fc640 10 monclient: dump: 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.746+0000 7fcad3fff640 4 set_mon_vals no callback set 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.748+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fcad801ad90 con 0x7fcadc0a9d70 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.748+0000 7fcae419d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.748+0000 7fcae419d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.748+0000 7fcae419d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:52.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.748+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac4005080 con 0x7fcadc0a9d70 2024-03-29T11:01:52.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.749+0000 7fcae1738640 10 monclient: get_auth_request con 0x7fcac006dbc0 auth_method 0 2024-03-29T11:01:52.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.749+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fcad8084df0 con 0x7fcadc0a9d70 2024-03-29T11:01:52.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.749+0000 7fcad27fc640 10 client.4409.objecter ms_dispatch 0x7fcadc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:52.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.749+0000 7fcad27fc640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:52.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.749+0000 7fcad27fc640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:52.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.749+0000 7fcad27fc640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:01:52.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.749+0000 7fcad27fc640 10 client.4409.objecter ms_handle_connect 0x7fcac006dbc0 2024-03-29T11:01:52.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.755+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcad8050730 con 0x7fcadc0a9d70 2024-03-29T11:01:52.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.755+0000 7fcad27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:52.755+0000 7fcad27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:53.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.049+0000 7fcae419d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:53.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.049+0000 7fcae419d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:53.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.049+0000 7fcae419d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:01:53.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.049+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcac4005330 con 0x7fcadc0a9d70 2024-03-29T11:01:53.029 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.050+0000 7fcad27fc640 1 -- 172.21.15.107:0/4259718165 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fcad8052660 con 0x7fcadc0a9d70 2024-03-29T11:01:53.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.050+0000 7fcad27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:53.030 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.050+0000 7fcad27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:53.030 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:01:53.030 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:53.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.053+0000 7fcae419d640 10 librados: watch_flush enter 2024-03-29T11:01:53.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.053+0000 7fcae419d640 10 librados: watch_flush exit 2024-03-29T11:01:53.033 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.053+0000 7fcae419d640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:01:53.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.053+0000 7fcae419d640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:01:53.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.053+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 >> v1:172.21.15.107:6816/882194755 conn(0x7fcac006dbc0 legacy=0x7fcac0070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:53.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.053+0000 7fcae419d640 10 monclient: shutdown 2024-03-29T11:01:53.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.053+0000 7fcae419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:53.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.053+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 >> v1:172.21.15.107:6790/0 conn(0x7fcadc0a9d70 legacy=0x7fcadc135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:53.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.054+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 shutdown_connections 2024-03-29T11:01:53.034 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.054+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 >> 172.21.15.107:0/4259718165 conn(0x7fcadc066a00 msgr2=0x7fcadc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:53.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.055+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 shutdown_connections 2024-03-29T11:01:53.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.055+0000 7fcae419d640 1 -- 172.21.15.107:0/4259718165 wait complete. 2024-03-29T11:01:53.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:01:53.055+0000 7fcae419d640 1 librados: shutdown 2024-03-29T11:01:53.050 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:59.857 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:59.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.005+0000 7fce2bd23640 10 monclient: get_monmap_and_config 2024-03-29T11:01:59.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.005+0000 7fce2bd23640 10 monclient: build_initial_monmap 2024-03-29T11:01:59.985 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.005+0000 7fce2bd23640 10 monclient: monmap: 2024-03-29T11:01:59.985 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:59.985 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:59.985 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:02:00.006531+0000 2024-03-29T11:01:59.986 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:02:00.006531+0000 2024-03-29T11:01:59.986 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:59.986 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:59.986 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:59.986 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:01:59.986 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:01:59.986 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.006+0000 7fce2bd23640 1 Processor -- start 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.006+0000 7fce2bd23640 1 -- start start 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.006+0000 7fce2bd23640 10 monclient: init 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 10 monclient(hunting): picked mon.noname-b con 0x7fce241422b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 10 monclient(hunting): picked mon.noname-a con 0x7fce24142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 10 monclient(hunting): picked mon.noname-c con 0x7fce240b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce24142c80 con 0x7fce24142690 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce29abf640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fce241422b0 0x7fce2413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40978/0 (socket says 172.21.15.107:40978) 2024-03-29T11:01:59.989 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce29abf640 1 -- 172.21.15.107:0/3644515802 learned_addr learned my addr 172.21.15.107:0/3644515802 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 1 -- 172.21.15.107:0/3644515802 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce240aae40 con 0x7fce240b17d0 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 1 -- 172.21.15.107:0/3644515802 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2413fbd0 con 0x7fce241422b0 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce2bd23640 10 monclient(hunting): authenticate will time out at 1051.275266s 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829879285 0 0) 0x7fce240aae40 con 0x7fce240b17d0 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce28abd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce28abd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce28abd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce28abd640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.007+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce10003520 con 0x7fce240b17d0 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.008+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104050750 0 0) 0x7fce2413fbd0 con 0x7fce241422b0 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.008+0000 7fce28abd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.008+0000 7fce28abd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.008+0000 7fce28abd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:59.990 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.008+0000 7fce28abd640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:01:59.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.008+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce240aae40 con 0x7fce241422b0 2024-03-29T11:01:59.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.008+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711146578 0 0) 0x7fce10003520 con 0x7fce240b17d0 2024-03-29T11:01:59.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.008+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 >> v1:172.21.15.107:6790/0 conn(0x7fce241422b0 legacy=0x7fce2413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:59.991 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.008+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 >> v1:172.21.15.107:6789/0 conn(0x7fce24142690 legacy=0x7fce240b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 1 monclient: found mon.noname-c 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce24151940 con 0x7fce240b17d0 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: _finish_auth 0 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fce1c001990 con 0x7fce240b17d0 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: dump: 2024-03-29T11:01:59.994 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce2bd23640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce2bd23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce1c001af0 con 0x7fce240b17d0 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 1 -- 172.21.15.107:0/3644515802 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fce1c001f00 con 0x7fce240b17d0 2024-03-29T11:01:59.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.010+0000 7fce28abd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.011+0000 7fce28abd640 10 monclient: dump: 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.011+0000 7fce2bd23640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.011+0000 7fce2bd23640 4 set_mon_vals no callback set 2024-03-29T11:01:59.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.012+0000 7fce2bd23640 10 monclient: shutdown 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.012+0000 7fce2bd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.012+0000 7fce2bd23640 1 -- 172.21.15.107:0/3644515802 >> v1:172.21.15.129:6789/0 conn(0x7fce240b17d0 legacy=0x7fce240b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.012+0000 7fce2bd23640 1 -- 172.21.15.107:0/3644515802 shutdown_connections 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.012+0000 7fce2bd23640 1 -- 172.21.15.107:0/3644515802 >> 172.21.15.107:0/3644515802 conn(0x7fce24066a00 msgr2=0x7fce24066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.012+0000 7fce2bd23640 1 -- 172.21.15.107:0/3644515802 shutdown_connections 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.012+0000 7fce2bd23640 1 -- 172.21.15.107:0/3644515802 wait complete. 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.013+0000 7fce2bd23640 10 monclient: build_initial_monmap 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.013+0000 7fce2bd23640 10 monclient: monmap: 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:01:59.997 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.013+0000 7fce2bd23640 1 librados: starting msgr at 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.013+0000 7fce2bd23640 1 librados: starting objecter 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.013+0000 7fce2bd23640 1 Processor -- start 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.013+0000 7fce2bd23640 1 -- start start 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.013+0000 7fce2bd23640 1 librados: setting wanted keys 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.013+0000 7fce2bd23640 1 librados: calling monclient init 2024-03-29T11:01:59.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.013+0000 7fce2bd23640 10 monclient: init 2024-03-29T11:02:00.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.014+0000 7fce2bd23640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:00.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.014+0000 7fce2bd23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:00.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce2bd23640 10 monclient(hunting): picked mon.noname-c con 0x7fce240b17d0 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:00.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce2bd23640 10 monclient(hunting): picked mon.noname-b con 0x7fce241422b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:00.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce2bd23640 10 monclient(hunting): picked mon.noname-a con 0x7fce24142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:00.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce2bd23640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce240b57f0 con 0x7fce24142690 2024-03-29T11:02:00.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce2bd23640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce240b5a60 con 0x7fce241422b0 2024-03-29T11:02:00.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce2bd23640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce240b5d40 con 0x7fce240b17d0 2024-03-29T11:02:00.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce2bd23640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce2bd23640 10 monclient(hunting): authenticate will time out at 1051.283058s 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce29abf640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fce240b17d0 0x7fce2414d6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41384/0 (socket says 172.21.15.107:41384) 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce292be640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fce241422b0 0x7fce24130a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:37022/0 (socket says 172.21.15.107:37022) 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce29abf640 1 -- 172.21.15.107:0/1032982778 learned_addr learned my addr 172.21.15.107:0/1032982778 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce1a7fc640 10 client.?.objecter ms_handle_connect 0x7fce240b17d0 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.015+0000 7fce1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336861066 0 0) 0x7fce240b5d40 con 0x7fce240b17d0 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce08004eb0 con 0x7fce240b17d0 2024-03-29T11:02:00.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 376505288 0 0) 0x7fce08004eb0 con 0x7fce240b17d0 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce240b5d40 con 0x7fce240b17d0 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fce200018f0 con 0x7fce240b17d0 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 10 monclient(hunting): dump: 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:00.004 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3550287061 0 0) 0x7fce240b5d40 con 0x7fce240b17d0 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 >> v1:172.21.15.129:6789/0 conn(0x7fce241422b0 legacy=0x7fce24130a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.016+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 >> v1:172.21.15.107:6789/0 conn(0x7fce24142690 legacy=0x7fce240b6340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce1a7fc640 1 monclient: found mon.c 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce1a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2414f3b0 con 0x7fce240b17d0 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce2bd23640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce2bd23640 10 monclient: _renew_subs 2024-03-29T11:02:00.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce2bd23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2414f660 con 0x7fce240b17d0 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce2bd23640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce2bd23640 10 monclient: _renew_subs 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce2bd23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2414dfb0 con 0x7fce240b17d0 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce2bd23640 1 librados: init done 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.017+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce20001c90 con 0x7fce240b17d0 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.018+0000 7fce1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.018+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fce20004c30 con 0x7fce240b17d0 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.018+0000 7fce1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.018+0000 7fce1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.018+0000 7fce1a7fc640 10 monclient: dump: 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:00.006 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.018+0000 7fce1bfff640 4 set_mon_vals no callback set 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.018+0000 7fce2bd23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.019+0000 7fce2bd23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.019+0000 7fce2bd23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.019+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c005080 con 0x7fce240b17d0 2024-03-29T11:02:00.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.019+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fce2001ac00 con 0x7fce240b17d0 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.020+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fce20084bf0 con 0x7fce240b17d0 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.020+0000 7fce1a7fc640 10 client.4415.objecter ms_dispatch 0x7fce24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.020+0000 7fce1a7fc640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.020+0000 7fce1a7fc640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.020+0000 7fce1a7fc640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.020+0000 7fce292be640 10 monclient: get_auth_request con 0x7fce0806dbf0 auth_method 0 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.021+0000 7fce1a7fc640 10 client.4415.objecter ms_handle_connect 0x7fce0806dbf0 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.026+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce20050300 con 0x7fce240b17d0 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.026+0000 7fce1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:00.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.026+0000 7fce1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:00.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.311+0000 7fce2bd23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:00.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.311+0000 7fce2bd23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:00.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.311+0000 7fce2bd23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:00.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.311+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce0c005330 con 0x7fce240b17d0 2024-03-29T11:02:00.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.312+0000 7fce1a7fc640 1 -- 172.21.15.107:0/1032982778 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fce20052400 con 0x7fce240b17d0 2024-03-29T11:02:00.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.312+0000 7fce1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:00.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.312+0000 7fce1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:00.292 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:02:00.293 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.315+0000 7fce2bd23640 10 librados: watch_flush enter 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.315+0000 7fce2bd23640 10 librados: watch_flush exit 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.315+0000 7fce2bd23640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.315+0000 7fce2bd23640 10 client.4415.objecter successfully canceled tick 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.315+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 >> v1:172.21.15.107:6816/882194755 conn(0x7fce0806dbf0 legacy=0x7fce080700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.315+0000 7fce2bd23640 10 monclient: shutdown 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.315+0000 7fce2bd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.315+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 >> v1:172.21.15.107:6790/0 conn(0x7fce240b17d0 legacy=0x7fce2414d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.316+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 shutdown_connections 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.316+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 >> 172.21.15.107:0/1032982778 conn(0x7fce24066a00 msgr2=0x7fce24132450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.316+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 shutdown_connections 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.316+0000 7fce2bd23640 1 -- 172.21.15.107:0/1032982778 wait complete. 2024-03-29T11:02:00.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:00.316+0000 7fce2bd23640 1 librados: shutdown 2024-03-29T11:02:00.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:07.115 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:07.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.283+0000 7fc0b8855640 10 monclient: get_monmap_and_config 2024-03-29T11:02:07.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.283+0000 7fc0b8855640 10 monclient: build_initial_monmap 2024-03-29T11:02:07.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.283+0000 7fc0b8855640 10 monclient: monmap: 2024-03-29T11:02:07.263 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:07.264 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:07.264 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:02:07.284613+0000 2024-03-29T11:02:07.264 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:02:07.284613+0000 2024-03-29T11:02:07.264 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:07.264 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:07.264 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:07.264 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:02:07.264 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:02:07.264 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:07.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.286+0000 7fc0b8855640 1 Processor -- start 2024-03-29T11:02:07.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.286+0000 7fc0b8855640 1 -- start start 2024-03-29T11:02:07.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.286+0000 7fc0b8855640 10 monclient: init 2024-03-29T11:02:07.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.287+0000 7fc0b8855640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:07.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.287+0000 7fc0b8855640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:07.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.287+0000 7fc0b8855640 10 monclient(hunting): picked mon.noname-c con 0x7fc0b40b4890 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.287+0000 7fc0b8855640 10 monclient(hunting): picked mon.noname-b con 0x7fc0b413c040 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b8855640 10 monclient(hunting): picked mon.noname-a con 0x7fc0b41427e0 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b8855640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b40b42f0 con 0x7fc0b41427e0 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b8855640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b40aa7a0 con 0x7fc0b40b4890 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b8855640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b40aaa10 con 0x7fc0b413c040 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b8855640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b8855640 10 monclient(hunting): authenticate will time out at 1058.557128s 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.287+0000 7fc0abfff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc0b413c040 0x7fc0b41420d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51030/0 (socket says 172.21.15.107:51030) 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0abfff640 1 -- 172.21.15.107:0/1549996673 learned_addr learned my addr 172.21.15.107:0/1549996673 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:07.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b1d9b640 1 -- 172.21.15.107:0/1549996673 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621663682 0 0) 0x7fc0b40aaa10 con 0x7fc0b413c040 2024-03-29T11:02:07.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b1d9b640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:02:07.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.289+0000 7fc0b1d9b640 1 -- 172.21.15.107:0/1549996673 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a0003520 con 0x7fc0b413c040 2024-03-29T11:02:07.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.290+0000 7fc0b1d9b640 1 -- 172.21.15.107:0/1549996673 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290004629 0 0) 0x7fc0a0003520 con 0x7fc0b413c040 2024-03-29T11:02:07.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.290+0000 7fc0b1d9b640 1 -- 172.21.15.107:0/1549996673 >> v1:172.21.15.129:6789/0 conn(0x7fc0b40b4890 legacy=0x7fc0b413b930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:07.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.290+0000 7fc0b1d9b640 1 -- 172.21.15.107:0/1549996673 >> v1:172.21.15.107:6789/0 conn(0x7fc0b41427e0 legacy=0x7fc0b414d880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 1 monclient: found mon.noname-b 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 1 -- 172.21.15.107:0/1549996673 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b40aacf0 con 0x7fc0b413c040 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 1 -- 172.21.15.107:0/1549996673 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc09c001690 con 0x7fc0b413c040 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:07.272 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b1d9b640 10 monclient: dump: 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b8855640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:02:07.273 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.292+0000 7fc0b8855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:07.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.293+0000 7fc0b1d9b640 1 -- 172.21.15.107:0/1549996673 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc09c0018f0 con 0x7fc0b413c040 2024-03-29T11:02:07.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.293+0000 7fc0b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:07.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.293+0000 7fc0b8855640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:07.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.293+0000 7fc0b8855640 4 set_mon_vals no callback set 2024-03-29T11:02:07.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.294+0000 7fc0b1d9b640 1 -- 172.21.15.107:0/1549996673 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc09c001cc0 con 0x7fc0b413c040 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.294+0000 7fc0b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.294+0000 7fc0b1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.294+0000 7fc0b1d9b640 10 monclient: dump: 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:07.276 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:07.277 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:07.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.294+0000 7fc0b8855640 10 monclient: shutdown 2024-03-29T11:02:07.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.294+0000 7fc0b8855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:07.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.294+0000 7fc0b8855640 1 -- 172.21.15.107:0/1549996673 >> v1:172.21.15.107:6790/0 conn(0x7fc0b413c040 legacy=0x7fc0b41420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.294+0000 7fc0b8855640 1 -- 172.21.15.107:0/1549996673 shutdown_connections 2024-03-29T11:02:07.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.294+0000 7fc0b8855640 1 -- 172.21.15.107:0/1549996673 >> 172.21.15.107:0/1549996673 conn(0x7fc0b406d180 msgr2=0x7fc0b4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:07.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.296+0000 7fc0b8855640 1 -- 172.21.15.107:0/1549996673 shutdown_connections 2024-03-29T11:02:07.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.296+0000 7fc0b8855640 1 -- 172.21.15.107:0/1549996673 wait complete. 2024-03-29T11:02:07.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.297+0000 7fc0b8855640 10 monclient: build_initial_monmap 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.297+0000 7fc0b8855640 10 monclient: monmap: 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.298+0000 7fc0b8855640 1 librados: starting msgr at 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.298+0000 7fc0b8855640 1 librados: starting objecter 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.298+0000 7fc0b8855640 1 Processor -- start 2024-03-29T11:02:07.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.298+0000 7fc0b8855640 1 -- start start 2024-03-29T11:02:07.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.298+0000 7fc0b8855640 1 librados: setting wanted keys 2024-03-29T11:02:07.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.298+0000 7fc0b8855640 1 librados: calling monclient init 2024-03-29T11:02:07.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.298+0000 7fc0b8855640 10 monclient: init 2024-03-29T11:02:07.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.299+0000 7fc0b8855640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:07.284 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.299+0000 7fc0b8855640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.299+0000 7fc0b8855640 10 monclient(hunting): picked mon.noname-c con 0x7fc0b40b4890 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.300+0000 7fc0b259c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc0b40b4890 0x7fc0b41da840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51698/0 (socket says 172.21.15.107:51698) 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.300+0000 7fc0b259c640 1 -- 172.21.15.107:0/4181883679 learned_addr learned my addr 172.21.15.107:0/4181883679 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.300+0000 7fc0b8855640 10 monclient(hunting): picked mon.noname-a con 0x7fc0b413c040 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0b8855640 10 monclient(hunting): picked mon.noname-b con 0x7fc0b41427e0 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0b8855640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b41da3b0 con 0x7fc0b413c040 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0b8855640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b413a7c0 con 0x7fc0b41427e0 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0b8855640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b41daf50 con 0x7fc0b40b4890 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0aaffd640 10 client.?.objecter ms_handle_connect 0x7fc0b40b4890 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0b8855640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0b8855640 10 monclient(hunting): authenticate will time out at 1058.569625s 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689195392 0 0) 0x7fc0b41daf50 con 0x7fc0b40b4890 2024-03-29T11:02:07.285 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0aaffd640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.301+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc094004eb0 con 0x7fc0b40b4890 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.302+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 347184148 0 0) 0x7fc094004eb0 con 0x7fc0b40b4890 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.302+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc094005300 con 0x7fc0b40b4890 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.302+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc0a4001830 con 0x7fc0b40b4890 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.302+0000 7fc0aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.302+0000 7fc0aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.302+0000 7fc0aaffd640 10 monclient(hunting): dump: 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:07.286 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.302+0000 7fc0aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.302+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2667619582 0 0) 0x7fc094005300 con 0x7fc0b40b4890 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.302+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 >> v1:172.21.15.129:6789/0 conn(0x7fc0b41427e0 legacy=0x7fc0b413a0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:07.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.304+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 >> v1:172.21.15.107:6789/0 conn(0x7fc0b413c040 legacy=0x7fc0b41399a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:07.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:07.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0aaffd640 1 monclient: found mon.c 2024-03-29T11:02:07.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:07.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:07.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b40abfc0 con 0x7fc0b40b4890 2024-03-29T11:02:07.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:02:07.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:07.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:07.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0b8855640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:02:07.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0b8855640 10 monclient: _renew_subs 2024-03-29T11:02:07.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0b8855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:07.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0b8855640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b40ac2a0 con 0x7fc0b40b4890 2024-03-29T11:02:07.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0b8855640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:07.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0b8855640 10 monclient: _renew_subs 2024-03-29T11:02:07.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0b8855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:07.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0b8855640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b413acb0 con 0x7fc0b40b4890 2024-03-29T11:02:07.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.306+0000 7fc0b8855640 1 librados: init done 2024-03-29T11:02:07.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.308+0000 7fc0a8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:07.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.308+0000 7fc0a8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:07.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.308+0000 7fc0a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:07.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.308+0000 7fc0a8ff9640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b41da110 con 0x7fc0b40b4890 2024-03-29T11:02:07.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.308+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0a4001e00 con 0x7fc0b40b4890 2024-03-29T11:02:07.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.308+0000 7fc0aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:07.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.308+0000 7fc0b159a640 4 set_mon_vals no callback set 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.309+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc0a4001830 con 0x7fc0b40b4890 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.309+0000 7fc0aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.309+0000 7fc0aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.309+0000 7fc0aaffd640 10 monclient: dump: 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:07.292 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:07.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.310+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fc0a401a8b0 con 0x7fc0b40b4890 2024-03-29T11:02:07.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.311+0000 7fc0abfff640 10 monclient: get_auth_request con 0x7fc09406e1d0 auth_method 0 2024-03-29T11:02:07.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.311+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fc0a4054460 con 0x7fc0b40b4890 2024-03-29T11:02:07.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.311+0000 7fc0aaffd640 10 client.4421.objecter ms_dispatch 0x7fc0b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:07.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.311+0000 7fc0aaffd640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:07.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.311+0000 7fc0aaffd640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:07.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.311+0000 7fc0aaffd640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:02:07.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.311+0000 7fc0aaffd640 10 client.4421.objecter ms_handle_connect 0x7fc09406e1d0 2024-03-29T11:02:07.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.320+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0a4050700 con 0x7fc0b40b4890 2024-03-29T11:02:07.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.320+0000 7fc0aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:07.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.320+0000 7fc0aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:07.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.636+0000 7fc0a8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.636+0000 7fc0a8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.636+0000 7fc0a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:07.616 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.636+0000 7fc0a8ff9640 1 -- 172.21.15.107:0/4181883679 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0b404f730 con 0x7fc0b40b4890 2024-03-29T11:02:07.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.638+0000 7fc0aaffd640 1 -- 172.21.15.107:0/4181883679 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fc0a4050700 con 0x7fc0b40b4890 2024-03-29T11:02:07.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.638+0000 7fc0aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.618 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.638+0000 7fc0aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:07.619 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:02:07.621 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:07.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.644+0000 7fc0a8ff9640 10 librados: watch_flush enter 2024-03-29T11:02:07.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 10 librados: watch_flush exit 2024-03-29T11:02:07.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:02:07.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:02:07.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 1 -- 172.21.15.107:0/4181883679 >> v1:172.21.15.107:6816/882194755 conn(0x7fc09406e1d0 legacy=0x7fc094070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 10 monclient: shutdown 2024-03-29T11:02:07.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:07.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 1 -- 172.21.15.107:0/4181883679 >> v1:172.21.15.107:6790/0 conn(0x7fc0b40b4890 legacy=0x7fc0b41da840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 1 -- 172.21.15.107:0/4181883679 shutdown_connections 2024-03-29T11:02:07.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 1 -- 172.21.15.107:0/4181883679 >> 172.21.15.107:0/4181883679 conn(0x7fc0b406d180 msgr2=0x7fc0b40b8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:07.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 1 -- 172.21.15.107:0/4181883679 shutdown_connections 2024-03-29T11:02:07.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.645+0000 7fc0a8ff9640 1 -- 172.21.15.107:0/4181883679 wait complete. 2024-03-29T11:02:07.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:07.646+0000 7fc0a8ff9640 1 librados: shutdown 2024-03-29T11:02:07.644 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:14.451 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:14.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.619+0000 7f472ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:02:14.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.619+0000 7f472ffff640 10 monclient: build_initial_monmap 2024-03-29T11:02:14.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.619+0000 7f472ffff640 10 monclient: monmap: 2024-03-29T11:02:14.600 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:14.600 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:14.600 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:02:14.620041+0000 2024-03-29T11:02:14.600 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:02:14.620041+0000 2024-03-29T11:02:14.600 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:14.600 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.620+0000 7f472ffff640 1 Processor -- start 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.620+0000 7f472ffff640 1 -- start start 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.620+0000 7f472ffff640 10 monclient: init 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:14.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f4730071f40 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:14.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f47300adcb0 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:14.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f473013f9a0 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:14.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47300af1b0 con 0x7f4730071f40 2024-03-29T11:02:14.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47300b0030 con 0x7f473013f9a0 2024-03-29T11:02:14.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47300aec40 con 0x7f47300adcb0 2024-03-29T11:02:14.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:14.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.621+0000 7f472ffff640 10 monclient(hunting): authenticate will time out at 1065.889450s 2024-03-29T11:02:14.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.624+0000 7f472e7fc640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f47300adcb0 0x7f47301457a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42770/0 (socket says 172.21.15.107:42770) 2024-03-29T11:02:14.604 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.624+0000 7f472e7fc640 1 -- 172.21.15.107:0/2086474655 learned_addr learned my addr 172.21.15.107:0/2086474655 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:14.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.625+0000 7f472dffb640 1 -- 172.21.15.107:0/2086474655 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158623995 0 0) 0x7f47300af1b0 con 0x7f4730071f40 2024-03-29T11:02:14.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.625+0000 7f472dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.625+0000 7f472dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.625+0000 7f472dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.625+0000 7f472dffb640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:02:14.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.625+0000 7f472dffb640 1 -- 172.21.15.107:0/2086474655 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4718003520 con 0x7f4730071f40 2024-03-29T11:02:14.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 1 -- 172.21.15.107:0/2086474655 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2218120229 0 0) 0x7f4718003520 con 0x7f4730071f40 2024-03-29T11:02:14.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 1 -- 172.21.15.107:0/2086474655 >> v1:172.21.15.107:6790/0 conn(0x7f47300adcb0 legacy=0x7f47301457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 1 -- 172.21.15.107:0/2086474655 >> v1:172.21.15.129:6789/0 conn(0x7f473013f9a0 legacy=0x7f4730145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:14.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:14.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 1 monclient: found mon.noname-a 2024-03-29T11:02:14.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:14.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:02:14.606 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 1 -- 172.21.15.107:0/2086474655 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47300aedd0 con 0x7f4730071f40 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 1 -- 172.21.15.107:0/2086474655 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4720001690 con 0x7f4730071f40 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472ffff640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: dump: 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.607 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 1 -- 172.21.15.107:0/2086474655 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4720001860 con 0x7f4730071f40 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 1 -- 172.21.15.107:0/2086474655 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4720001cb0 con 0x7f4730071f40 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472ffff640 4 set_mon_vals no callback set 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.626+0000 7f472dffb640 10 monclient: dump: 2024-03-29T11:02:14.608 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:14.609 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:14.609 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:14.609 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:14.609 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.609 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:14.609 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:14.609 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:14.609 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:14.609 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:14.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.627+0000 7f472ffff640 10 monclient: shutdown 2024-03-29T11:02:14.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.627+0000 7f472ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:14.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.627+0000 7f472ffff640 1 -- 172.21.15.107:0/2086474655 >> v1:172.21.15.107:6789/0 conn(0x7f4730071f40 legacy=0x7f473013f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.627+0000 7f472ffff640 1 -- 172.21.15.107:0/2086474655 shutdown_connections 2024-03-29T11:02:14.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.627+0000 7f472ffff640 1 -- 172.21.15.107:0/2086474655 >> 172.21.15.107:0/2086474655 conn(0x7f47300673e0 msgr2=0x7f47300b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:14.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.630+0000 7f472ffff640 1 -- 172.21.15.107:0/2086474655 shutdown_connections 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.630+0000 7f472ffff640 1 -- 172.21.15.107:0/2086474655 wait complete. 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.631+0000 7f472ffff640 10 monclient: build_initial_monmap 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.631+0000 7f472ffff640 10 monclient: monmap: 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:02:14.611 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:14.613 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.631+0000 7f472ffff640 1 librados: starting msgr at 2024-03-29T11:02:14.613 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.631+0000 7f472ffff640 1 librados: starting objecter 2024-03-29T11:02:14.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.632+0000 7f472ffff640 1 Processor -- start 2024-03-29T11:02:14.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.632+0000 7f472ffff640 1 -- start start 2024-03-29T11:02:14.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.632+0000 7f472ffff640 1 librados: setting wanted keys 2024-03-29T11:02:14.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.632+0000 7f472ffff640 1 librados: calling monclient init 2024-03-29T11:02:14.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.632+0000 7f472ffff640 10 monclient: init 2024-03-29T11:02:14.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:14.614 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:14.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f47300adcb0 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:14.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f473013f9a0 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:14.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f4730140180 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:14.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4730133390 con 0x7f4730140180 2024-03-29T11:02:14.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47301dba80 con 0x7f473013f9a0 2024-03-29T11:02:14.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47301dbc10 con 0x7f47300adcb0 2024-03-29T11:02:14.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:14.615 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.633+0000 7f472ffff640 10 monclient(hunting): authenticate will time out at 1065.901270s 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f472effd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f47300adcb0 0x7f47301db370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43612/0 (socket says 172.21.15.107:43612) 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f472effd640 1 -- 172.21.15.107:0/2930394530 learned_addr learned my addr 172.21.15.107:0/2930394530 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f47177fe640 10 client.?.objecter ms_handle_connect 0x7f47300adcb0 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f47177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820058882 0 0) 0x7f47301dbc10 con 0x7f47300adcb0 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f47177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f47177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f47177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f47177fe640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.635+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f471c004eb0 con 0x7f47300adcb0 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 client.?.objecter ms_handle_connect 0x7f473013f9a0 2024-03-29T11:02:14.622 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2152071716 0 0) 0x7f471c004eb0 con 0x7f47300adcb0 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47301dbc10 con 0x7f47300adcb0 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4720001b70 con 0x7f47300adcb0 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient(hunting): dump: 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:14.623 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849443275 0 0) 0x7f47301dba80 con 0x7f473013f9a0 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f471c004eb0 con 0x7f473013f9a0 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2098031340 0 0) 0x7f47301dbc10 con 0x7f47300adcb0 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 >> v1:172.21.15.129:6789/0 conn(0x7f473013f9a0 legacy=0x7f4730132570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 >> v1:172.21.15.107:6789/0 conn(0x7f4730140180 legacy=0x7f4730132c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:14.624 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 monclient: found mon.c 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4730141560 con 0x7f47300adcb0 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f47177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f472ffff640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f472ffff640 10 monclient: _renew_subs 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f472ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:14.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.636+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4730141750 con 0x7f47300adcb0 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f472ffff640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f472ffff640 10 monclient: _renew_subs 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f472ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47301407e0 con 0x7f47300adcb0 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f472ffff640 1 librados: init done 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4720003960 con 0x7f47300adcb0 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f47177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4720003d30 con 0x7f47300adcb0 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f47177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f47177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:14.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.637+0000 7f47177fe640 10 monclient: dump: 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.638+0000 7f472d7fa640 4 set_mon_vals no callback set 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.638+0000 7f472ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:14.627 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.639+0000 7f472ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.639+0000 7f472ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.639+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4710005080 con 0x7f47300adcb0 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.639+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f4720003e90 con 0x7f47300adcb0 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.640+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f4720054910 con 0x7f47300adcb0 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.640+0000 7f47177fe640 10 client.4427.objecter ms_dispatch 0x7f4730000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.640+0000 7f47177fe640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.640+0000 7f47177fe640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.640+0000 7f47177fe640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.640+0000 7f472e7fc640 10 monclient: get_auth_request con 0x7f471c06dd00 auth_method 0 2024-03-29T11:02:14.628 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.641+0000 7f47177fe640 10 client.4427.objecter ms_handle_connect 0x7f471c06dd00 2024-03-29T11:02:14.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.649+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4720050bb0 con 0x7f47300adcb0 2024-03-29T11:02:14.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.649+0000 7f47177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:14.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.649+0000 7f47177fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:14.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.969+0000 7f472ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:14.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.969+0000 7f472ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:14.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.969+0000 7f472ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:14.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.969+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4710005330 con 0x7f47300adcb0 2024-03-29T11:02:14.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.971+0000 7f47177fe640 1 -- 172.21.15.107:0/2930394530 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f4720050bb0 con 0x7f47300adcb0 2024-03-29T11:02:14.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.971+0000 7f47177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:14.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.971+0000 7f47177fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:14.951 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:02:14.951 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:14.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.974+0000 7f472ffff640 10 librados: watch_flush enter 2024-03-29T11:02:14.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.975+0000 7f472ffff640 10 librados: watch_flush exit 2024-03-29T11:02:14.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.975+0000 7f472ffff640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:02:14.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.975+0000 7f472ffff640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:02:14.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.976+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 >> v1:172.21.15.107:6816/882194755 conn(0x7f471c06dd00 legacy=0x7f471c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.976+0000 7f472ffff640 10 monclient: shutdown 2024-03-29T11:02:14.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.976+0000 7f472ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:14.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.976+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 >> v1:172.21.15.107:6790/0 conn(0x7f47300adcb0 legacy=0x7f47301db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.976+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 shutdown_connections 2024-03-29T11:02:14.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.976+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 >> 172.21.15.107:0/2930394530 conn(0x7f47300673e0 msgr2=0x7f4730069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:14.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.976+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 shutdown_connections 2024-03-29T11:02:14.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.976+0000 7f472ffff640 1 -- 172.21.15.107:0/2930394530 wait complete. 2024-03-29T11:02:14.958 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:14.978+0000 7f472ffff640 1 librados: shutdown 2024-03-29T11:02:14.976 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:21.783 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:21.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.959+0000 7fc22c915640 10 monclient: get_monmap_and_config 2024-03-29T11:02:21.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.959+0000 7fc22c915640 10 monclient: build_initial_monmap 2024-03-29T11:02:21.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.959+0000 7fc22c915640 10 monclient: monmap: 2024-03-29T11:02:21.939 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:21.940 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:21.940 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:02:21.960914+0000 2024-03-29T11:02:21.940 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:02:21.960914+0000 2024-03-29T11:02:21.940 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:21.940 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:21.940 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:21.940 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:02:21.940 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:02:21.940 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:21.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.961+0000 7fc22c915640 1 Processor -- start 2024-03-29T11:02:21.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.962+0000 7fc22c915640 1 -- start start 2024-03-29T11:02:21.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.962+0000 7fc22c915640 10 monclient: init 2024-03-29T11:02:21.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:21.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:21.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 10 monclient(hunting): picked mon.noname-c con 0x7fc2281457f0 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:21.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 10 monclient(hunting): picked mon.noname-a con 0x7fc22813f650 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:21.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 10 monclient(hunting): picked mon.noname-b con 0x7fc2280a9d70 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:21.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2280aaef0 con 0x7fc22813f650 2024-03-29T11:02:21.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc228145bd0 con 0x7fc2281457f0 2024-03-29T11:02:21.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc228145eb0 con 0x7fc2280a9d70 2024-03-29T11:02:21.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:21.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.965+0000 7fc22c915640 10 monclient(hunting): authenticate will time out at 1073.233218s 2024-03-29T11:02:21.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.968+0000 7fc22659c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc2281457f0 0x7fc22813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:48955/0 (socket says 172.21.15.107:48955) 2024-03-29T11:02:21.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.968+0000 7fc22659c640 1 -- 172.21.15.107:0/1760198099 learned_addr learned my addr 172.21.15.107:0/1760198099 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:21.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002602270 0 0) 0x7fc228145eb0 con 0x7fc2280a9d70 2024-03-29T11:02:21.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:21.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:21.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:21.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:02:21.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc214003520 con 0x7fc2280a9d70 2024-03-29T11:02:21.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484043429 0 0) 0x7fc228145bd0 con 0x7fc2281457f0 2024-03-29T11:02:21.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:21.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:21.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:21.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:02:21.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc228145eb0 con 0x7fc2281457f0 2024-03-29T11:02:21.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 959704660 0 0) 0x7fc228145eb0 con 0x7fc2281457f0 2024-03-29T11:02:21.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 >> v1:172.21.15.107:6790/0 conn(0x7fc2280a9d70 legacy=0x7fc22814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:21.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.969+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 >> v1:172.21.15.107:6789/0 conn(0x7fc22813f650 legacy=0x7fc2280b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.971+0000 7fc22559a640 10 monclient: _finish_hunting 0 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.971+0000 7fc22559a640 1 monclient: found mon.noname-c 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.971+0000 7fc22559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.971+0000 7fc22559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.971+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc228146340 con 0x7fc2281457f0 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 10 monclient: _finish_auth 0 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 10 monclient: _check_auth_tickets 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc2180019f0 con 0x7fc2281457f0 2024-03-29T11:02:21.954 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 10 monclient: dump: 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc218001c10 con 0x7fc2281457f0 2024-03-29T11:02:21.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 1 -- 172.21.15.107:0/1760198099 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc218001fe0 con 0x7fc2281457f0 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22559a640 10 monclient: dump: 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:21.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22c915640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:02:21.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22c915640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:21.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.972+0000 7fc22c915640 4 set_mon_vals no callback set 2024-03-29T11:02:21.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.974+0000 7fc22c915640 10 monclient: shutdown 2024-03-29T11:02:21.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.974+0000 7fc22c915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:21.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.974+0000 7fc22c915640 1 -- 172.21.15.107:0/1760198099 >> v1:172.21.15.129:6789/0 conn(0x7fc2281457f0 legacy=0x7fc22813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:21.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.974+0000 7fc22c915640 1 -- 172.21.15.107:0/1760198099 shutdown_connections 2024-03-29T11:02:21.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.974+0000 7fc22c915640 1 -- 172.21.15.107:0/1760198099 >> 172.21.15.107:0/1760198099 conn(0x7fc228066a00 msgr2=0x7fc228066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:21.957 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.974+0000 7fc22c915640 1 -- 172.21.15.107:0/1760198099 shutdown_connections 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.974+0000 7fc22c915640 1 -- 172.21.15.107:0/1760198099 wait complete. 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.975+0000 7fc22c915640 10 monclient: build_initial_monmap 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.975+0000 7fc22c915640 10 monclient: monmap: 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:21.970 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.976+0000 7fc22c915640 1 librados: starting msgr at 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.976+0000 7fc22c915640 1 librados: starting objecter 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.976+0000 7fc22c915640 1 Processor -- start 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.976+0000 7fc22c915640 1 -- start start 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.976+0000 7fc22c915640 1 librados: setting wanted keys 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.976+0000 7fc22c915640 1 librados: calling monclient init 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.976+0000 7fc22c915640 10 monclient: init 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.977+0000 7fc22c915640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.977+0000 7fc22c915640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:21.971 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.977+0000 7fc22c915640 10 monclient(hunting): picked mon.noname-a con 0x7fc2280a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.980+0000 7fc22c915640 10 monclient(hunting): picked mon.noname-b con 0x7fc22813f650 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.980+0000 7fc22c915640 10 monclient(hunting): picked mon.noname-c con 0x7fc22806de90 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.980+0000 7fc22c915640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2281424e0 con 0x7fc2280a9d70 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.980+0000 7fc22c915640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22806f140 con 0x7fc22813f650 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.980+0000 7fc226d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc22806de90 0x7fc2281e6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:37790/0 (socket says 172.21.15.107:37790) 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.980+0000 7fc226d9d640 1 -- 172.21.15.107:0/308175688 learned_addr learned my addr 172.21.15.107:0/308175688 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.980+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc228144ed0 con 0x7fc22806de90 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc22c915640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc22c915640 10 monclient(hunting): authenticate will time out at 1073.249140s 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 10 client.?.objecter ms_handle_connect 0x7fc22806de90 2024-03-29T11:02:21.972 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 10 client.?.objecter ms_handle_connect 0x7fc2280a9d70 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043804439 0 0) 0x7fc228144ed0 con 0x7fc22806de90 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc208004eb0 con 0x7fc22806de90 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932935686 0 0) 0x7fc2281424e0 con 0x7fc2280a9d70 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.981+0000 7fc21effd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:21.973 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc228144ed0 con 0x7fc2280a9d70 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2734819903 0 0) 0x7fc208004eb0 con 0x7fc22806de90 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2281424e0 con 0x7fc22806de90 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc210001990 con 0x7fc22806de90 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 10 monclient(hunting): dump: 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:21.974 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2870810017 0 0) 0x7fc228144ed0 con 0x7fc2280a9d70 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc208004eb0 con 0x7fc2280a9d70 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2336480189 0 0) 0x7fc2281424e0 con 0x7fc22806de90 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.982+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 >> v1:172.21.15.129:6789/0 conn(0x7fc22813f650 legacy=0x7fc228137450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.983+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 >> v1:172.21.15.107:6789/0 conn(0x7fc2280a9d70 legacy=0x7fc228141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.983+0000 7fc21effd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.983+0000 7fc21effd640 1 monclient: found mon.c 2024-03-29T11:02:21.975 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.983+0000 7fc21effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.983+0000 7fc21effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.983+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2281449b0 con 0x7fc22806de90 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.983+0000 7fc21effd640 10 monclient: _finish_auth 0 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.983+0000 7fc21effd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.983+0000 7fc21effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc210001af0 con 0x7fc22806de90 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc22c915640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc22c915640 10 monclient: _renew_subs 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc22c915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc228144c00 con 0x7fc22806de90 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc22c915640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc22c915640 10 monclient: _renew_subs 2024-03-29T11:02:21.976 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc22c915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22813ff80 con 0x7fc22806de90 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc22c915640 1 librados: init done 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc21effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc2100038c0 con 0x7fc22806de90 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc21effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc21effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.984+0000 7fc21effd640 10 monclient: dump: 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:21.977 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.986+0000 7fc224d99640 4 set_mon_vals no callback set 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.986+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fc210003a20 con 0x7fc22806de90 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.987+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fc210084a00 con 0x7fc22806de90 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.990+0000 7fc22c915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.990+0000 7fc21effd640 10 client.4433.objecter ms_dispatch 0x7fc228000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.990+0000 7fc21effd640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.990+0000 7fc21effd640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.990+0000 7fc21effd640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.987+0000 7fc22659c640 10 monclient: get_auth_request con 0x7fc20806de60 auth_method 0 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.990+0000 7fc22c915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.990+0000 7fc22c915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:21.978 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.990+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20c005080 con 0x7fc22806de90 2024-03-29T11:02:21.979 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:21.990+0000 7fc21effd640 10 client.4433.objecter ms_handle_connect 0x7fc20806de60 2024-03-29T11:02:21.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.000+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc210050110 con 0x7fc22806de90 2024-03-29T11:02:21.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.000+0000 7fc21effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:21.980 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.000+0000 7fc21effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:22.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.322+0000 7fc22c915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:22.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.322+0000 7fc22c915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:22.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.322+0000 7fc22c915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:22.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.322+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc20c005330 con 0x7fc22806de90 2024-03-29T11:02:22.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.323+0000 7fc21effd640 1 -- 172.21.15.107:0/308175688 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fc210052210 con 0x7fc22806de90 2024-03-29T11:02:22.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.324+0000 7fc21effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:22.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.324+0000 7fc21effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:22.304 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:02:22.304 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:22.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.327+0000 7fc22c915640 10 librados: watch_flush enter 2024-03-29T11:02:22.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.327+0000 7fc22c915640 10 librados: watch_flush exit 2024-03-29T11:02:22.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.327+0000 7fc22c915640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-29T11:02:22.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.327+0000 7fc22c915640 10 client.4433.objecter successfully canceled tick 2024-03-29T11:02:22.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.328+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 >> v1:172.21.15.107:6816/882194755 conn(0x7fc20806de60 legacy=0x7fc208070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:22.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.328+0000 7fc22c915640 10 monclient: shutdown 2024-03-29T11:02:22.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.328+0000 7fc22c915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:22.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.328+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 >> v1:172.21.15.107:6790/0 conn(0x7fc22806de90 legacy=0x7fc2281e6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:22.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.328+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 shutdown_connections 2024-03-29T11:02:22.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.328+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 >> 172.21.15.107:0/308175688 conn(0x7fc228066a00 msgr2=0x7fc2280b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:22.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.329+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 shutdown_connections 2024-03-29T11:02:22.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.329+0000 7fc22c915640 1 -- 172.21.15.107:0/308175688 wait complete. 2024-03-29T11:02:22.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:22.329+0000 7fc22c915640 1 librados: shutdown 2024-03-29T11:02:22.326 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:29.133 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:29.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.295+0000 7fccd9f52640 10 monclient: get_monmap_and_config 2024-03-29T11:02:29.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.295+0000 7fccd9f52640 10 monclient: build_initial_monmap 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.295+0000 7fccd9f52640 10 monclient: monmap: 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:02:29.295992+0000 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:02:29.295992+0000 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:02:29.275 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:29.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.298+0000 7fccd9f52640 1 Processor -- start 2024-03-29T11:02:29.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.298+0000 7fccd9f52640 1 -- start start 2024-03-29T11:02:29.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.298+0000 7fccd9f52640 10 monclient: init 2024-03-29T11:02:29.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:29.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:29.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 10 monclient(hunting): picked mon.noname-b con 0x7fccd41422b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:29.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 10 monclient(hunting): picked mon.noname-c con 0x7fccd4142690 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:29.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 10 monclient(hunting): picked mon.noname-a con 0x7fccd40b17d0 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:29.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd4142c80 con 0x7fccd40b17d0 2024-03-29T11:02:29.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd40aae40 con 0x7fccd4142690 2024-03-29T11:02:29.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd413fbd0 con 0x7fccd41422b0 2024-03-29T11:02:29.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:29.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd9f52640 10 monclient(hunting): authenticate will time out at 1080.567246s 2024-03-29T11:02:29.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd37fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fccd41422b0 0x7fccd413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34892/0 (socket says 172.21.15.107:34892) 2024-03-29T11:02:29.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.299+0000 7fccd37fe640 1 -- 172.21.15.107:0/52507066 learned_addr learned my addr 172.21.15.107:0/52507066 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:29.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.302+0000 7fccd27fc640 1 -- 172.21.15.107:0/52507066 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120531669 0 0) 0x7fccd413fbd0 con 0x7fccd41422b0 2024-03-29T11:02:29.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.302+0000 7fccd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:29.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.302+0000 7fccd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:29.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.302+0000 7fccd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:29.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.302+0000 7fccd27fc640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:02:29.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.302+0000 7fccd27fc640 1 -- 172.21.15.107:0/52507066 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccbc003520 con 0x7fccd41422b0 2024-03-29T11:02:29.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.302+0000 7fccd27fc640 1 -- 172.21.15.107:0/52507066 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2095000698 0 0) 0x7fccbc003520 con 0x7fccd41422b0 2024-03-29T11:02:29.282 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.302+0000 7fccd27fc640 1 -- 172.21.15.107:0/52507066 >> v1:172.21.15.129:6789/0 conn(0x7fccd4142690 legacy=0x7fccd40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:29.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.305+0000 7fccd27fc640 1 -- 172.21.15.107:0/52507066 >> v1:172.21.15.107:6789/0 conn(0x7fccd40b17d0 legacy=0x7fccd40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.305+0000 7fccd27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:02:29.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.305+0000 7fccd27fc640 1 monclient: found mon.noname-b 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.305+0000 7fccd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.305+0000 7fccd27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 1 -- 172.21.15.107:0/52507066 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4151940 con 0x7fccd41422b0 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 10 monclient: _finish_auth 0 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 1 -- 172.21.15.107:0/52507066 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fccc8001690 con 0x7fccd41422b0 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 10 monclient: dump: 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:29.287 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd9f52640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd9f52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 1 -- 172.21.15.107:0/52507066 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccc80018f0 con 0x7fccd41422b0 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd9f52640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.306+0000 7fccd9f52640 4 set_mon_vals no callback set 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.307+0000 7fccd27fc640 1 -- 172.21.15.107:0/52507066 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fccc8001c80 con 0x7fccd41422b0 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.307+0000 7fccd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.307+0000 7fccd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:29.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.307+0000 7fccd27fc640 10 monclient: dump: 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:29.289 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:29.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.307+0000 7fccd9f52640 10 monclient: shutdown 2024-03-29T11:02:29.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.307+0000 7fccd9f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:29.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.307+0000 7fccd9f52640 1 -- 172.21.15.107:0/52507066 >> v1:172.21.15.107:6790/0 conn(0x7fccd41422b0 legacy=0x7fccd413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.290 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.308+0000 7fccd9f52640 1 -- 172.21.15.107:0/52507066 shutdown_connections 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.308+0000 7fccd9f52640 1 -- 172.21.15.107:0/52507066 >> 172.21.15.107:0/52507066 conn(0x7fccd4066a00 msgr2=0x7fccd4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.309+0000 7fccd9f52640 1 -- 172.21.15.107:0/52507066 shutdown_connections 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.309+0000 7fccd9f52640 1 -- 172.21.15.107:0/52507066 wait complete. 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.310+0000 7fccd9f52640 10 monclient: build_initial_monmap 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.310+0000 7fccd9f52640 10 monclient: monmap: 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:02:29.291 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:02:29.292 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:29.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.311+0000 7fccd9f52640 1 librados: starting msgr at 2024-03-29T11:02:29.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.311+0000 7fccd9f52640 1 librados: starting objecter 2024-03-29T11:02:29.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.311+0000 7fccd9f52640 1 Processor -- start 2024-03-29T11:02:29.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.311+0000 7fccd9f52640 1 -- start start 2024-03-29T11:02:29.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.311+0000 7fccd9f52640 1 librados: setting wanted keys 2024-03-29T11:02:29.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.311+0000 7fccd9f52640 1 librados: calling monclient init 2024-03-29T11:02:29.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.311+0000 7fccd9f52640 10 monclient: init 2024-03-29T11:02:29.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:29.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:29.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 10 monclient(hunting): picked mon.noname-a con 0x7fccd40b17d0 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:29.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 10 monclient(hunting): picked mon.noname-b con 0x7fccd41422b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:29.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 10 monclient(hunting): picked mon.noname-c con 0x7fccd4142690 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:29.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd414e300 con 0x7fccd40b17d0 2024-03-29T11:02:29.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd40acd60 con 0x7fccd41422b0 2024-03-29T11:02:29.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd40ad040 con 0x7fccd4142690 2024-03-29T11:02:29.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:29.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.312+0000 7fccd9f52640 10 monclient(hunting): authenticate will time out at 1080.580511s 2024-03-29T11:02:29.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.314+0000 7fccd3fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fccd4142690 0x7fccd414dbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:36490/0 (socket says 172.21.15.107:36490) 2024-03-29T11:02:29.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.314+0000 7fccd3fff640 1 -- 172.21.15.107:0/3794261204 learned_addr learned my addr 172.21.15.107:0/3794261204 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:29.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.314+0000 7fccbbfff640 10 client.?.objecter ms_handle_connect 0x7fccd4142690 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.314+0000 7fccbbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.315+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308730093 0 0) 0x7fccd40ad040 con 0x7fccd4142690 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.315+0000 7fccbbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.315+0000 7fccbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.315+0000 7fccbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.315+0000 7fccbbfff640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.315+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc0004eb0 con 0x7fccd4142690 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.315+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 878325823 0 0) 0x7fccc0004eb0 con 0x7fccd4142690 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.316+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccd40ad040 con 0x7fccd4142690 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.316+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcccc001c30 con 0x7fccd4142690 2024-03-29T11:02:29.296 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.316+0000 7fccbbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.316+0000 7fccbbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.316+0000 7fccbbfff640 10 monclient(hunting): dump: 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:29.297 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:29.298 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:29.298 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:29.298 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.316+0000 7fccbbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:29.302 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.317+0000 7fccbbfff640 10 client.?.objecter ms_handle_connect 0x7fccd40b17d0 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 client.?.objecter ms_handle_connect 0x7fccd41422b0 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443413461 0 0) 0x7fccd414e300 con 0x7fccd40b17d0 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc0004eb0 con 0x7fccd40b17d0 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426951589 0 0) 0x7fccd40acd60 con 0x7fccd41422b0 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:29.303 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd414e300 con 0x7fccd41422b0 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1483141228 0 0) 0x7fccd40ad040 con 0x7fccd4142690 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 >> v1:172.21.15.129:6789/0 conn(0x7fccd41422b0 legacy=0x7fccd414d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 >> v1:172.21.15.107:6789/0 conn(0x7fccd40b17d0 legacy=0x7fccd41da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 1 monclient: found mon.c 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd40ab6e0 con 0x7fccd4142690 2024-03-29T11:02:29.304 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient: _finish_auth 0 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccd9f52640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccd9f52640 10 monclient: _renew_subs 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccd9f52640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.318+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd41db0a0 con 0x7fccd4142690 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.319+0000 7fccd9f52640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.319+0000 7fccd9f52640 10 monclient: _renew_subs 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.319+0000 7fccd9f52640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.319+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd40b6d50 con 0x7fccd4142690 2024-03-29T11:02:29.305 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.319+0000 7fccd9f52640 1 librados: init done 2024-03-29T11:02:29.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.321+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcccc001d90 con 0x7fccd4142690 2024-03-29T11:02:29.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.321+0000 7fccbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:29.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.321+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fcccc003b30 con 0x7fccd4142690 2024-03-29T11:02:29.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.321+0000 7fccbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:29.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.321+0000 7fccbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:29.306 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.321+0000 7fccbbfff640 10 monclient: dump: 2024-03-29T11:02:29.306 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:29.306 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:29.306 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.321+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fcccc003d10 con 0x7fccd4142690 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.321+0000 7fccd1ffb640 4 set_mon_vals no callback set 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.322+0000 7fccd37fe640 10 monclient: get_auth_request con 0x7fccc006de60 auth_method 0 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.322+0000 7fccbbfff640 10 client.4439.objecter ms_handle_connect 0x7fccc006de60 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.322+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fcccc054d60 con 0x7fccd4142690 2024-03-29T11:02:29.307 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.322+0000 7fccbbfff640 10 client.4439.objecter ms_dispatch 0x7fccd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:29.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.322+0000 7fccbbfff640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:29.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.322+0000 7fccbbfff640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:29.308 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.322+0000 7fccbbfff640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:02:29.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.323+0000 7fccd9f52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:29.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.323+0000 7fccd9f52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:29.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.323+0000 7fccd9f52640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:29.309 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.323+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb4005080 con 0x7fccd4142690 2024-03-29T11:02:29.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.333+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcccc0506a0 con 0x7fccd4142690 2024-03-29T11:02:29.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.333+0000 7fccbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:29.313 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.333+0000 7fccbbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:29.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.658+0000 7fccd9f52640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:29.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.658+0000 7fccd9f52640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:29.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.658+0000 7fccd9f52640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:02:29.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.658+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb4005330 con 0x7fccd4142690 2024-03-29T11:02:29.638 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.659+0000 7fccbbfff640 1 -- 172.21.15.107:0/3794261204 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fcccc0527a0 con 0x7fccd4142690 2024-03-29T11:02:29.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.659+0000 7fccbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:29.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.659+0000 7fccbbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:29.639 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:02:29.639 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:29.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.662+0000 7fccd9f52640 10 librados: watch_flush enter 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.663+0000 7fccd9f52640 10 librados: watch_flush exit 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.663+0000 7fccd9f52640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.663+0000 7fccd9f52640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.663+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 >> v1:172.21.15.107:6816/882194755 conn(0x7fccc006de60 legacy=0x7fccc0070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.663+0000 7fccd9f52640 10 monclient: shutdown 2024-03-29T11:02:29.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.663+0000 7fccd9f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.663+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 >> v1:172.21.15.107:6790/0 conn(0x7fccd4142690 legacy=0x7fccd414dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.665+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 shutdown_connections 2024-03-29T11:02:29.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.665+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 >> 172.21.15.107:0/3794261204 conn(0x7fccd4066a00 msgr2=0x7fccd413c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.666+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 shutdown_connections 2024-03-29T11:02:29.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.666+0000 7fccd9f52640 1 -- 172.21.15.107:0/3794261204 wait complete. 2024-03-29T11:02:29.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:29.667+0000 7fccd9f52640 1 librados: shutdown 2024-03-29T11:02:29.663 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:36.469 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:36.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.644+0000 7f8257060640 10 monclient: get_monmap_and_config 2024-03-29T11:02:36.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.644+0000 7f8257060640 10 monclient: build_initial_monmap 2024-03-29T11:02:36.625 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.644+0000 7f8257060640 10 monclient: monmap: 2024-03-29T11:02:36.625 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:02:36.645660+0000 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:02:36.645660+0000 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.646+0000 7f8257060640 1 Processor -- start 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.646+0000 7f8257060640 1 -- start start 2024-03-29T11:02:36.626 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.646+0000 7f8257060640 10 monclient: init 2024-03-29T11:02:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.646+0000 7f8257060640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.646+0000 7f8257060640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.646+0000 7f8257060640 10 monclient(hunting): picked mon.noname-a con 0x7f8250071c30 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.646+0000 7f8257060640 10 monclient(hunting): picked mon.noname-c con 0x7f82500a9750 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.646+0000 7f8257060640 10 monclient(hunting): picked mon.noname-b con 0x7f825013e1e0 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f8257060640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500ab300 con 0x7f8250071c30 2024-03-29T11:02:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f8257060640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500ac2f0 con 0x7f82500a9750 2024-03-29T11:02:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f8257060640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82500ad8d0 con 0x7f825013e1e0 2024-03-29T11:02:36.629 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f8257060640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f8257060640 10 monclient(hunting): authenticate will time out at 1087.915228s 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f8254dfc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8250071c30 0x7f82500b0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59300/0 (socket says 172.21.15.107:59300) 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f8254dfc640 1 -- 172.21.15.107:0/865329132 learned_addr learned my addr 172.21.15.107:0/865329132 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460287125 0 0) 0x7f82500ab300 con 0x7f8250071c30 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f824f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f824f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f824f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f824f7fe640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8238003520 con 0x7f8250071c30 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.647+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794299643 0 0) 0x7f82500ad8d0 con 0x7f825013e1e0 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:36.630 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82500ab300 con 0x7f825013e1e0 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663836926 0 0) 0x7f82500ac2f0 con 0x7f82500a9750 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82500ad8d0 con 0x7f82500a9750 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2919774840 0 0) 0x7f82500ab300 con 0x7f825013e1e0 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 >> v1:172.21.15.129:6789/0 conn(0x7f82500a9750 legacy=0x7f825013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 >> v1:172.21.15.107:6789/0 conn(0x7f8250071c30 legacy=0x7f82500b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:36.631 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82500ada60 con 0x7f825013e1e0 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f8257060640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f8257060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f82480019f0 con 0x7f825013e1e0 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f824f7fe640 10 monclient: dump: 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.632 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.648+0000 7f8257060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.649+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8248001b50 con 0x7f825013e1e0 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.649+0000 7f824f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.649+0000 7f824f7fe640 1 -- 172.21.15.107:0/865329132 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f8248001f60 con 0x7f825013e1e0 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.649+0000 7f824f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.649+0000 7f824f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.649+0000 7f824f7fe640 10 monclient: dump: 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:36.633 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.649+0000 7f8257060640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:36.634 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.649+0000 7f8257060640 4 set_mon_vals no callback set 2024-03-29T11:02:36.639 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.650+0000 7f8257060640 10 monclient: shutdown 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.650+0000 7f8257060640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.650+0000 7f8257060640 1 -- 172.21.15.107:0/865329132 >> v1:172.21.15.107:6790/0 conn(0x7f825013e1e0 legacy=0x7f82500ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.650+0000 7f8257060640 1 -- 172.21.15.107:0/865329132 shutdown_connections 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.650+0000 7f8257060640 1 -- 172.21.15.107:0/865329132 >> 172.21.15.107:0/865329132 conn(0x7f825006b660 msgr2=0x7f825006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.650+0000 7f8257060640 1 -- 172.21.15.107:0/865329132 shutdown_connections 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.650+0000 7f8257060640 1 -- 172.21.15.107:0/865329132 wait complete. 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.652+0000 7f8257060640 10 monclient: build_initial_monmap 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.652+0000 7f8257060640 10 monclient: monmap: 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:02:36.640 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.652+0000 7f8257060640 1 librados: starting msgr at 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.652+0000 7f8257060640 1 librados: starting objecter 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.652+0000 7f8257060640 1 Processor -- start 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.652+0000 7f8257060640 1 -- start start 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.652+0000 7f8257060640 1 librados: setting wanted keys 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.652+0000 7f8257060640 1 librados: calling monclient init 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.652+0000 7f8257060640 10 monclient: init 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:36.641 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 10 monclient(hunting): picked mon.noname-c con 0x7f8250071c30 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 10 monclient(hunting): picked mon.noname-a con 0x7f82500a9750 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 10 monclient(hunting): picked mon.noname-b con 0x7f825013e1e0 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825013fa40 con 0x7f82500a9750 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82501426f0 con 0x7f825013e1e0 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8250142880 con 0x7f8250071c30 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.653+0000 7f8257060640 10 monclient(hunting): authenticate will time out at 1087.921940s 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f8254dfc640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f8250071c30 0x7f825013efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38888/0 (socket says 172.21.15.107:38888) 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f8254dfc640 1 -- 172.21.15.107:0/23909934 learned_addr learned my addr 172.21.15.107:0/23909934 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 client.?.objecter ms_handle_connect 0x7f825013e1e0 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:36.642 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 client.?.objecter ms_handle_connect 0x7f8250071c30 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 client.?.objecter ms_handle_connect 0x7f82500a9750 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232310405 0 0) 0x7f82501426f0 con 0x7f825013e1e0 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.654+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f823c004eb0 con 0x7f825013e1e0 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818510962 0 0) 0x7f825013fa40 con 0x7f82500a9750 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:02:36.643 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82501426f0 con 0x7f82500a9750 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263489009 0 0) 0x7f8250142880 con 0x7f8250071c30 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825013fa40 con 0x7f8250071c30 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1452814199 0 0) 0x7f823c004eb0 con 0x7f825013e1e0 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8250142880 con 0x7f825013e1e0 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f8248001e60 con 0x7f825013e1e0 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): dump: 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:36.644 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.655+0000 7f824d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 744821744 0 0) 0x7f82501426f0 con 0x7f82500a9750 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f823c004eb0 con 0x7f82500a9750 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f82440018f0 con 0x7f82500a9750 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:36.645 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient(hunting): dump: 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1362307851 0 0) 0x7f8250142880 con 0x7f825013e1e0 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 >> v1:172.21.15.107:6790/0 conn(0x7f8250071c30 legacy=0x7f825013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 >> v1:172.21.15.107:6789/0 conn(0x7f82500a9750 legacy=0x7f82501da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:36.646 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 monclient: found mon.b 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8250142a10 con 0x7f825013e1e0 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f8257060640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f8257060640 10 monclient: _renew_subs 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f8257060640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82501dae90 con 0x7f825013e1e0 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f8257060640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f8257060640 10 monclient: _renew_subs 2024-03-29T11:02:36.647 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f8257060640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f825013e600 con 0x7f825013e1e0 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f8257060640 1 librados: init done 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8248001550 con 0x7f825013e1e0 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824effd640 4 set_mon_vals no callback set 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f8248003c20 con 0x7f825013e1e0 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.656+0000 7f824d7fa640 10 monclient: dump: 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:36.648 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.658+0000 7f8257060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.658+0000 7f8257060640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.658+0000 7f8257060640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.658+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8230005080 con 0x7f825013e1e0 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.658+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f8248019c70 con 0x7f825013e1e0 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.659+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f8248083bf0 con 0x7f825013e1e0 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.659+0000 7f824d7fa640 10 client.4390.objecter ms_dispatch 0x7f8250000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.659+0000 7f824d7fa640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.659+0000 7f824d7fa640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:36.649 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.659+0000 7f824d7fa640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T11:02:36.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.659+0000 7f8254dfc640 10 monclient: get_auth_request con 0x7f823c06ddb0 auth_method 0 2024-03-29T11:02:36.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.660+0000 7f824d7fa640 10 client.4390.objecter ms_handle_connect 0x7f823c06ddb0 2024-03-29T11:02:36.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.668+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f824804f300 con 0x7f825013e1e0 2024-03-29T11:02:36.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.668+0000 7f824d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:36.650 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:36.668+0000 7f824d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:37.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.034+0000 7f8257060640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:37.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.034+0000 7f8257060640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:37.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.034+0000 7f8257060640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:02:37.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.034+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8230005330 con 0x7f825013e1e0 2024-03-29T11:02:37.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.036+0000 7f824d7fa640 1 -- 172.21.15.107:0/23909934 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f8248051400 con 0x7f825013e1e0 2024-03-29T11:02:37.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.036+0000 7f824d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:37.016 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.036+0000 7f824d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:37.016 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:02:37.016 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:37.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.039+0000 7f8257060640 10 librados: watch_flush enter 2024-03-29T11:02:37.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.039+0000 7f8257060640 10 librados: watch_flush exit 2024-03-29T11:02:37.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.039+0000 7f8257060640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T11:02:37.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.039+0000 7f8257060640 10 client.4390.objecter successfully canceled tick 2024-03-29T11:02:37.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.040+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 >> v1:172.21.15.107:6816/882194755 conn(0x7f823c06ddb0 legacy=0x7f823c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:37.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.040+0000 7f8257060640 10 monclient: shutdown 2024-03-29T11:02:37.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.040+0000 7f8257060640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:37.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.040+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 >> v1:172.21.15.129:6789/0 conn(0x7f825013e1e0 legacy=0x7f82501e1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:37.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.040+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 shutdown_connections 2024-03-29T11:02:37.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.040+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 >> 172.21.15.107:0/23909934 conn(0x7f825006b660 msgr2=0x7f82500b04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:37.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.040+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 shutdown_connections 2024-03-29T11:02:37.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.040+0000 7f8257060640 1 -- 172.21.15.107:0/23909934 wait complete. 2024-03-29T11:02:37.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:37.040+0000 7f8257060640 1 librados: shutdown 2024-03-29T11:02:37.035 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:43.840 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:43.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.012+0000 7fabff119640 10 monclient: get_monmap_and_config 2024-03-29T11:02:43.992 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.012+0000 7fabff119640 10 monclient: build_initial_monmap 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.012+0000 7fabff119640 10 monclient: monmap: 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:02:44.013769+0000 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:02:44.013769+0000 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:02:43.993 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:43.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.013+0000 7fabff119640 1 Processor -- start 2024-03-29T11:02:43.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.013+0000 7fabff119640 1 -- start start 2024-03-29T11:02:43.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.013+0000 7fabff119640 10 monclient: init 2024-03-29T11:02:43.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:43.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:43.995 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 10 monclient(hunting): picked mon.noname-b con 0x7fabf80ae490 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:43.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 10 monclient(hunting): picked mon.noname-c con 0x7fabf80aef70 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:43.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 10 monclient(hunting): picked mon.noname-a con 0x7fabf80727a0 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:43.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf80b0020 con 0x7fabf80727a0 2024-03-29T11:02:43.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf806d960 con 0x7fabf80aef70 2024-03-29T11:02:43.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf80ad310 con 0x7fabf80ae490 2024-03-29T11:02:43.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:43.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.014+0000 7fabff119640 10 monclient(hunting): authenticate will time out at 1095.282834s 2024-03-29T11:02:43.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.015+0000 7fabfe117640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fabf80ae490 0x7fabf80713a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60406/0 (socket says 172.21.15.107:60406) 2024-03-29T11:02:43.996 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.015+0000 7fabfe117640 1 -- 172.21.15.107:0/1368210251 learned_addr learned my addr 172.21.15.107:0/1368210251 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:43.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687421211 0 0) 0x7fabf80b0020 con 0x7fabf80727a0 2024-03-29T11:02:43.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 10 monclient(hunting): _negotiate 2024-03-29T11:02:43.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:43.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:43.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:02:43.997 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe4003520 con 0x7fabf80727a0 2024-03-29T11:02:43.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736439933 0 0) 0x7fabf80ad310 con 0x7fabf80ae490 2024-03-29T11:02:43.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 10 monclient(hunting): _negotiate 2024-03-29T11:02:43.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:43.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:43.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:02:43.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf80b0020 con 0x7fabf80ae490 2024-03-29T11:02:43.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1813842335 0 0) 0x7fabe4003520 con 0x7fabf80727a0 2024-03-29T11:02:43.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 >> v1:172.21.15.107:6790/0 conn(0x7fabf80ae490 legacy=0x7fabf80713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:43.998 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.016+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 >> v1:172.21.15.129:6789/0 conn(0x7fabf80aef70 legacy=0x7fabf8072090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.018+0000 7fabfd115640 10 monclient: _finish_hunting 0 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.018+0000 7fabfd115640 1 monclient: found mon.noname-a 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf81432d0 con 0x7fabf80727a0 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 10 monclient: _finish_auth 0 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 10 monclient: _check_auth_tickets 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabff119640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:02:44.001 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabff119640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fabf0001a10 con 0x7fabf80727a0 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 10 monclient: dump: 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:44.002 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabff119640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabf0001c30 con 0x7fabf80727a0 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.019+0000 7fabfd115640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.020+0000 7fabff119640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.020+0000 7fabff119640 4 set_mon_vals no callback set 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.020+0000 7fabfd115640 1 -- 172.21.15.107:0/1368210251 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fabf0001fc0 con 0x7fabf80727a0 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.020+0000 7fabfd115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.020+0000 7fabfd115640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.020+0000 7fabfd115640 10 monclient: dump: 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:44.003 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.020+0000 7fabff119640 10 monclient: shutdown 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.020+0000 7fabff119640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.020+0000 7fabff119640 1 -- 172.21.15.107:0/1368210251 >> v1:172.21.15.107:6789/0 conn(0x7fabf80727a0 legacy=0x7fabf814b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.021+0000 7fabff119640 1 -- 172.21.15.107:0/1368210251 shutdown_connections 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.021+0000 7fabff119640 1 -- 172.21.15.107:0/1368210251 >> 172.21.15.107:0/1368210251 conn(0x7fabf80673e0 msgr2=0x7fabf80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.021+0000 7fabff119640 1 -- 172.21.15.107:0/1368210251 shutdown_connections 2024-03-29T11:02:44.004 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.021+0000 7fabff119640 1 -- 172.21.15.107:0/1368210251 wait complete. 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.022+0000 7fabff119640 10 monclient: build_initial_monmap 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.022+0000 7fabff119640 10 monclient: monmap: 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:44.005 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.022+0000 7fabff119640 1 librados: starting msgr at 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.022+0000 7fabff119640 1 librados: starting objecter 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.023+0000 7fabff119640 1 Processor -- start 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.023+0000 7fabff119640 1 -- start start 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.023+0000 7fabff119640 1 librados: setting wanted keys 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.023+0000 7fabff119640 1 librados: calling monclient init 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.023+0000 7fabff119640 10 monclient: init 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 10 monclient(hunting): picked mon.noname-b con 0x7fabf80727a0 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 10 monclient(hunting): picked mon.noname-c con 0x7fabf80ae490 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 10 monclient(hunting): picked mon.noname-a con 0x7fabf80aef70 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:44.006 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf811c5f0 con 0x7fabf80aef70 2024-03-29T11:02:44.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf813c650 con 0x7fabf80727a0 2024-03-29T11:02:44.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf811c8d0 con 0x7fabf80ae490 2024-03-29T11:02:44.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:44.007 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabff119640 10 monclient(hunting): authenticate will time out at 1095.292235s 2024-03-29T11:02:44.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabfd916640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fabf80ae490 0x7fabf813b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60586/0 (socket says 172.21.15.107:60586) 2024-03-29T11:02:44.008 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabfd916640 1 -- 172.21.15.107:0/2203729769 learned_addr learned my addr 172.21.15.107:0/2203729769 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.024+0000 7fabeeffd640 10 client.?.objecter ms_handle_connect 0x7fabf80ae490 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 client.?.objecter ms_handle_connect 0x7fabf80aef70 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073536650 0 0) 0x7fabf811c5f0 con 0x7fabf80aef70 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabdc004eb0 con 0x7fabf80aef70 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891011943 0 0) 0x7fabf811c8d0 con 0x7fabf80ae490 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:44.009 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.025+0000 7fabeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf811c5f0 con 0x7fabf80ae490 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2313490986 0 0) 0x7fabdc004eb0 con 0x7fabf80aef70 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabf811c8d0 con 0x7fabf80aef70 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fabf0001950 con 0x7fabf80aef70 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 10 monclient(hunting): dump: 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:44.010 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3679677784 0 0) 0x7fabf811c8d0 con 0x7fabf80aef70 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 >> v1:172.21.15.107:6790/0 conn(0x7fabf80ae490 legacy=0x7fabf813b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 >> v1:172.21.15.129:6789/0 conn(0x7fabf80727a0 legacy=0x7fabf81de7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.026+0000 7fabeeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabeeffd640 1 monclient: found mon.a 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabeeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:44.011 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf811b4a0 con 0x7fabf80aef70 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabeeffd640 10 monclient: _finish_auth 0 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabeeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabff119640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabff119640 10 monclient: _renew_subs 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabff119640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf811b9c0 con 0x7fabf80aef70 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabff119640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabff119640 10 monclient: _renew_subs 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabff119640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf80acac0 con 0x7fabf80aef70 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.027+0000 7fabff119640 1 librados: init done 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabff119640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabf0003c20 con 0x7fabf80aef70 2024-03-29T11:02:44.012 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabeeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fabf0001ab0 con 0x7fabf80aef70 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabeeffd640 10 monclient: dump: 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:44.013 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:44.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabff119640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:44.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabff119640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:44.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.028+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf80acd70 con 0x7fabf80aef70 2024-03-29T11:02:44.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.029+0000 7fabfc914640 4 set_mon_vals no callback set 2024-03-29T11:02:44.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.031+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fabf0019da0 con 0x7fabf80aef70 2024-03-29T11:02:44.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.031+0000 7fabfe117640 10 monclient: get_auth_request con 0x7fabdc06dd00 auth_method 0 2024-03-29T11:02:44.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.032+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fabf0083c00 con 0x7fabf80aef70 2024-03-29T11:02:44.014 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.032+0000 7fabeeffd640 10 client.4407.objecter ms_dispatch 0x7fabf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:44.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.032+0000 7fabeeffd640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:44.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.032+0000 7fabeeffd640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:44.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.032+0000 7fabeeffd640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T11:02:44.015 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.032+0000 7fabeeffd640 10 client.4407.objecter ms_handle_connect 0x7fabdc06dd00 2024-03-29T11:02:44.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.041+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabf004f310 con 0x7fabf80aef70 2024-03-29T11:02:44.020 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.041+0000 7fabeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:44.021 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.041+0000 7fabeeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:44.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.405+0000 7fabff119640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:44.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.405+0000 7fabff119640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:44.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.405+0000 7fabff119640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:44.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.405+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabf80ae870 con 0x7fabf80aef70 2024-03-29T11:02:44.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.409+0000 7fabeeffd640 1 -- 172.21.15.107:0/2203729769 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fabf0051410 con 0x7fabf80aef70 2024-03-29T11:02:44.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.409+0000 7fabeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:44.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.409+0000 7fabeeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:44.390 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:02:44.390 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:44.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.414+0000 7fabff119640 10 librados: watch_flush enter 2024-03-29T11:02:44.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.414+0000 7fabff119640 10 librados: watch_flush exit 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.414+0000 7fabff119640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.414+0000 7fabff119640 10 client.4407.objecter successfully canceled tick 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.414+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 >> v1:172.21.15.107:6816/882194755 conn(0x7fabdc06dd00 legacy=0x7fabdc0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.414+0000 7fabff119640 10 monclient: shutdown 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.414+0000 7fabff119640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.414+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 >> v1:172.21.15.107:6789/0 conn(0x7fabf80aef70 legacy=0x7fabf813bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.416+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 shutdown_connections 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.416+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 >> 172.21.15.107:0/2203729769 conn(0x7fabf80673e0 msgr2=0x7fabf8072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.416+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 shutdown_connections 2024-03-29T11:02:44.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.416+0000 7fabff119640 1 -- 172.21.15.107:0/2203729769 wait complete. 2024-03-29T11:02:44.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:44.416+0000 7fabff119640 1 librados: shutdown 2024-03-29T11:02:44.412 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:51.221 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:51.370 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.391+0000 7feb72446640 10 monclient: get_monmap_and_config 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.391+0000 7feb72446640 10 monclient: build_initial_monmap 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.391+0000 7feb72446640 10 monclient: monmap: 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:02:51.392026+0000 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:02:51.392026+0000 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:02:51.371 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:51.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.391+0000 7feb72446640 1 Processor -- start 2024-03-29T11:02:51.372 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.392+0000 7feb72446640 1 -- start start 2024-03-29T11:02:51.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.392+0000 7feb72446640 10 monclient: init 2024-03-29T11:02:51.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.393+0000 7feb72446640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:51.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.393+0000 7feb72446640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:51.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.393+0000 7feb72446640 10 monclient(hunting): picked mon.noname-a con 0x7feb640389e0 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:51.373 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.393+0000 7feb72446640 10 monclient(hunting): picked mon.noname-b con 0x7feb64038dc0 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:51.375 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.396+0000 7feb72446640 10 monclient(hunting): picked mon.noname-c con 0x7feb64030660 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:51.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.397+0000 7feb71444640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feb640389e0 0x7feb64033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45477/0 (socket says 172.21.15.107:45477) 2024-03-29T11:02:51.379 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.397+0000 7feb71444640 1 -- 172.21.15.107:0/2471338686 learned_addr learned my addr 172.21.15.107:0/2471338686 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.400+0000 7feb72446640 1 -- 172.21.15.107:0/2471338686 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb640c54c0 con 0x7feb640389e0 2024-03-29T11:02:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.400+0000 7feb72446640 1 -- 172.21.15.107:0/2471338686 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb64036430 con 0x7feb64030660 2024-03-29T11:02:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.400+0000 7feb72446640 1 -- 172.21.15.107:0/2471338686 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb64039840 con 0x7feb64038dc0 2024-03-29T11:02:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.400+0000 7feb72446640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:51.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.400+0000 7feb72446640 10 monclient(hunting): authenticate will time out at 1102.668956s 2024-03-29T11:02:51.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693097453 0 0) 0x7feb640c54c0 con 0x7feb640389e0 2024-03-29T11:02:51.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:51.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:51.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:51.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:02:51.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb58003520 con 0x7feb640389e0 2024-03-29T11:02:51.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022791623 0 0) 0x7feb64036430 con 0x7feb64030660 2024-03-29T11:02:51.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:51.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.404+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb640c54c0 con 0x7feb64030660 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.405+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107932185 0 0) 0x7feb64039840 con 0x7feb64038dc0 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.405+0000 7feb63fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.405+0000 7feb63fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.405+0000 7feb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.405+0000 7feb63fff640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.405+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb64036430 con 0x7feb64038dc0 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 32713521 0 0) 0x7feb640c54c0 con 0x7feb64030660 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 >> v1:172.21.15.107:6790/0 conn(0x7feb64038dc0 legacy=0x7feb640cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 >> v1:172.21.15.107:6789/0 conn(0x7feb640389e0 legacy=0x7feb64033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 1 monclient: found mon.noname-c 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb64035690 con 0x7feb64030660 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 10 monclient: _finish_auth 0 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7feb6c04f730 con 0x7feb64030660 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.406+0000 7feb63fff640 10 monclient: dump: 2024-03-29T11:02:51.390 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb72446640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb72446640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:51.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb6c075f10 con 0x7feb64030660 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb63fff640 1 -- 172.21.15.107:0/2471338686 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7feb6c0758b0 con 0x7feb64030660 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb63fff640 10 monclient: dump: 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:51.392 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:51.393 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:51.393 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:51.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb72446640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:51.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.407+0000 7feb72446640 4 set_mon_vals no callback set 2024-03-29T11:02:51.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.408+0000 7feb72446640 10 monclient: shutdown 2024-03-29T11:02:51.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.408+0000 7feb72446640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:51.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.408+0000 7feb72446640 1 -- 172.21.15.107:0/2471338686 >> v1:172.21.15.129:6789/0 conn(0x7feb64030660 legacy=0x7feb640d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.409+0000 7feb72446640 1 -- 172.21.15.107:0/2471338686 shutdown_connections 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.409+0000 7feb72446640 1 -- 172.21.15.107:0/2471338686 >> 172.21.15.107:0/2471338686 conn(0x7feb64005c00 msgr2=0x7feb64006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.409+0000 7feb72446640 1 -- 172.21.15.107:0/2471338686 shutdown_connections 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.409+0000 7feb72446640 1 -- 172.21.15.107:0/2471338686 wait complete. 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.410+0000 7feb72446640 10 monclient: build_initial_monmap 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.410+0000 7feb72446640 10 monclient: monmap: 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:02:51.395 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.411+0000 7feb72446640 1 librados: starting msgr at 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.411+0000 7feb72446640 1 librados: starting objecter 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.411+0000 7feb72446640 1 Processor -- start 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.411+0000 7feb72446640 1 -- start start 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.411+0000 7feb72446640 1 librados: setting wanted keys 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.411+0000 7feb72446640 1 librados: calling monclient init 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.412+0000 7feb72446640 10 monclient: init 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.412+0000 7feb72446640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:51.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.412+0000 7feb72446640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.415+0000 7feb72446640 10 monclient(hunting): picked mon.noname-a con 0x7feb64030660 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.415+0000 7feb72446640 10 monclient(hunting): picked mon.noname-c con 0x7feb640389e0 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.415+0000 7feb72446640 10 monclient(hunting): picked mon.noname-b con 0x7feb640c7670 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.415+0000 7feb72446640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6415c980 con 0x7feb64030660 2024-03-29T11:02:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.415+0000 7feb72446640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb640c72c0 con 0x7feb640c7670 2024-03-29T11:02:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.415+0000 7feb72446640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6415d520 con 0x7feb640389e0 2024-03-29T11:02:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.415+0000 7feb72446640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:51.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.415+0000 7feb72446640 10 monclient(hunting): authenticate will time out at 1102.683417s 2024-03-29T11:02:51.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.419+0000 7feb71444640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feb64030660 0x7feb6402fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47389/0 (socket says 172.21.15.107:47389) 2024-03-29T11:02:51.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.419+0000 7feb71444640 1 -- 172.21.15.107:0/1667352492 learned_addr learned my addr 172.21.15.107:0/1667352492 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:51.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.419+0000 7feb61ffb640 10 client.?.objecter ms_handle_connect 0x7feb64030660 2024-03-29T11:02:51.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.419+0000 7feb61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:51.400 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.419+0000 7feb61ffb640 10 client.?.objecter ms_handle_connect 0x7feb640389e0 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.419+0000 7feb61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458126182 0 0) 0x7feb6415c980 con 0x7feb64030660 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb50004eb0 con 0x7feb64030660 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1672400553 0 0) 0x7feb50004eb0 con 0x7feb64030660 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb6415c980 con 0x7feb64030660 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7feb68001990 con 0x7feb64030660 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 10 monclient(hunting): dump: 2024-03-29T11:02:51.401 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.420+0000 7feb61ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:51.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.421+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3109135638 0 0) 0x7feb6415c980 con 0x7feb64030660 2024-03-29T11:02:51.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 >> v1:172.21.15.107:6790/0 conn(0x7feb640389e0 legacy=0x7feb640c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 >> v1:172.21.15.129:6789/0 conn(0x7feb640c7670 legacy=0x7feb6415ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:51.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 1 monclient: found mon.a 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6415d6b0 con 0x7feb64030660 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb72446640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb72446640 10 monclient: _renew_subs 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb72446640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb72446640 1 -- 172.21.15.107:0/1667352492 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb640c7dd0 con 0x7feb64030660 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.422+0000 7feb72446640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.423+0000 7feb72446640 10 monclient: _renew_subs 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.423+0000 7feb72446640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:51.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.423+0000 7feb72446640 1 -- 172.21.15.107:0/1667352492 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb640c8350 con 0x7feb64030660 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.423+0000 7feb72446640 1 librados: init done 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.424+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb680047e0 con 0x7feb64030660 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.424+0000 7feb61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.424+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7feb68004bf0 con 0x7feb64030660 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.424+0000 7feb61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.424+0000 7feb61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.424+0000 7feb61ffb640 10 monclient: dump: 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:51.406 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:51.407 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:51.407 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:51.407 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:51.407 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.424+0000 7feb637fe640 4 set_mon_vals no callback set 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.425+0000 7feb577fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.425+0000 7feb577fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.425+0000 7feb577fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.425+0000 7feb577fe640 1 -- 172.21.15.107:0/1667352492 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb64007b90 con 0x7feb64030660 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.426+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7feb68004e10 con 0x7feb64030660 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.427+0000 7feb70c43640 10 monclient: get_auth_request con 0x7feb5006dbe0 auth_method 0 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.427+0000 7feb61ffb640 10 client.4413.objecter ms_handle_connect 0x7feb5006dbe0 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.428+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7feb68054980 con 0x7feb64030660 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.428+0000 7feb61ffb640 10 client.4413.objecter ms_dispatch 0x7feb6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.428+0000 7feb61ffb640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:51.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.428+0000 7feb61ffb640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:51.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.428+0000 7feb61ffb640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T11:02:51.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.435+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb68050c20 con 0x7feb64030660 2024-03-29T11:02:51.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.435+0000 7feb61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:51.414 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.435+0000 7feb61ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:51.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.751+0000 7feb577fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:51.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.751+0000 7feb577fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:51.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.751+0000 7feb577fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:51.731 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.751+0000 7feb577fe640 1 -- 172.21.15.107:0/1667352492 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb640c7f60 con 0x7feb64030660 2024-03-29T11:02:51.733 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.754+0000 7feb61ffb640 1 -- 172.21.15.107:0/1667352492 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7feb68050c20 con 0x7feb64030660 2024-03-29T11:02:51.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.754+0000 7feb61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:51.734 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.754+0000 7feb61ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:51.734 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:02:51.734 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:51.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.757+0000 7feb577fe640 10 librados: watch_flush enter 2024-03-29T11:02:51.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.757+0000 7feb577fe640 10 librados: watch_flush exit 2024-03-29T11:02:51.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.757+0000 7feb577fe640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T11:02:51.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.757+0000 7feb577fe640 10 client.4413.objecter successfully canceled tick 2024-03-29T11:02:51.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.758+0000 7feb577fe640 1 -- 172.21.15.107:0/1667352492 >> v1:172.21.15.107:6816/882194755 conn(0x7feb5006dbe0 legacy=0x7feb500700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.758+0000 7feb577fe640 10 monclient: shutdown 2024-03-29T11:02:51.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.758+0000 7feb577fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:51.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.758+0000 7feb577fe640 1 -- 172.21.15.107:0/1667352492 >> v1:172.21.15.107:6789/0 conn(0x7feb64030660 legacy=0x7feb6402fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.758+0000 7feb577fe640 1 -- 172.21.15.107:0/1667352492 shutdown_connections 2024-03-29T11:02:51.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.758+0000 7feb577fe640 1 -- 172.21.15.107:0/1667352492 >> 172.21.15.107:0/1667352492 conn(0x7feb64005c00 msgr2=0x7feb64034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:51.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.759+0000 7feb577fe640 1 -- 172.21.15.107:0/1667352492 shutdown_connections 2024-03-29T11:02:51.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.759+0000 7feb577fe640 1 -- 172.21.15.107:0/1667352492 wait complete. 2024-03-29T11:02:51.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:51.759+0000 7feb577fe640 1 librados: shutdown 2024-03-29T11:02:51.756 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:58.564 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.730+0000 7fbb3976c640 10 monclient: get_monmap_and_config 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.730+0000 7fbb3976c640 10 monclient: build_initial_monmap 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.730+0000 7fbb3976c640 10 monclient: monmap: 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:02:58.731188+0000 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:02:58.731188+0000 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:58.711 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.731+0000 7fbb3976c640 1 Processor -- start 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.731+0000 7fbb3976c640 1 -- start start 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.731+0000 7fbb3976c640 10 monclient: init 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 10 monclient(hunting): picked mon.noname-c con 0x7fbb340b4100 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:58.712 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 10 monclient(hunting): picked mon.noname-a con 0x7fbb3413ee90 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:58.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 10 monclient(hunting): picked mon.noname-b con 0x7fbb340a9ac0 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:58.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb340af6f0 con 0x7fbb3413ee90 2024-03-29T11:02:58.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb340aac80 con 0x7fbb340b4100 2024-03-29T11:02:58.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb340aaf60 con 0x7fbb340a9ac0 2024-03-29T11:02:58.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:58.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.732+0000 7fbb3976c640 10 monclient(hunting): authenticate will time out at 1110.000314s 2024-03-29T11:02:58.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb32ffd640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fbb340b4100 0x7fbb340adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:52246/0 (socket says 172.21.15.107:52246) 2024-03-29T11:02:58.715 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb32ffd640 1 -- 172.21.15.107:0/1564307562 learned_addr learned my addr 172.21.15.107:0/1564307562 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:58.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652309444 0 0) 0x7fbb340aac80 con 0x7fbb340b4100 2024-03-29T11:02:58.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:58.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:58.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:58.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:02:58.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb18003520 con 0x7fbb340b4100 2024-03-29T11:02:58.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104732768 0 0) 0x7fbb340aaf60 con 0x7fbb340a9ac0 2024-03-29T11:02:58.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:58.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:58.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:58.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:02:58.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb180048a0 con 0x7fbb340a9ac0 2024-03-29T11:02:58.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622933098 0 0) 0x7fbb340af6f0 con 0x7fbb3413ee90 2024-03-29T11:02:58.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:58.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:58.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:58.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:02:58.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.736+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb18004be0 con 0x7fbb3413ee90 2024-03-29T11:02:58.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1811056486 0 0) 0x7fbb180048a0 con 0x7fbb340a9ac0 2024-03-29T11:02:58.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 >> v1:172.21.15.129:6789/0 conn(0x7fbb340b4100 legacy=0x7fbb340adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:58.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 >> v1:172.21.15.107:6789/0 conn(0x7fbb3413ee90 legacy=0x7fbb3413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 1 monclient: found mon.noname-b 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3413e970 con 0x7fbb340a9ac0 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb3976c640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb3976c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:58.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbb2c001970 con 0x7fbb340a9ac0 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 10 monclient: dump: 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:58.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb3976c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb2c001b50 con 0x7fbb340a9ac0 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb3976c640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.737+0000 7fbb3976c640 4 set_mon_vals no callback set 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.738+0000 7fbb3976c640 10 monclient: shutdown 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.738+0000 7fbb3976c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.738+0000 7fbb3976c640 1 -- 172.21.15.107:0/1564307562 >> v1:172.21.15.107:6790/0 conn(0x7fbb340a9ac0 legacy=0x7fbb34142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.738+0000 7fbb31ffb640 1 -- 172.21.15.107:0/1564307562 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbb2c001970 con 0x7fbb340a9ac0 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.738+0000 7fbb31ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.739+0000 7fbb3976c640 1 -- 172.21.15.107:0/1564307562 shutdown_connections 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.739+0000 7fbb3976c640 1 -- 172.21.15.107:0/1564307562 >> 172.21.15.107:0/1564307562 conn(0x7fbb340b35e0 msgr2=0x7fbb340b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.739+0000 7fbb3976c640 1 -- 172.21.15.107:0/1564307562 shutdown_connections 2024-03-29T11:02:58.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.739+0000 7fbb3976c640 1 -- 172.21.15.107:0/1564307562 wait complete. 2024-03-29T11:02:58.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.740+0000 7fbb3976c640 10 monclient: build_initial_monmap 2024-03-29T11:02:58.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.740+0000 7fbb3976c640 10 monclient: monmap: 2024-03-29T11:02:58.735 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:02:58.735 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.740+0000 7fbb3976c640 1 librados: starting msgr at 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.740+0000 7fbb3976c640 1 librados: starting objecter 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.741+0000 7fbb3976c640 1 Processor -- start 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.741+0000 7fbb3976c640 1 -- start start 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.741+0000 7fbb3976c640 1 librados: setting wanted keys 2024-03-29T11:02:58.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.741+0000 7fbb3976c640 1 librados: calling monclient init 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.741+0000 7fbb3976c640 10 monclient: init 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 10 monclient(hunting): picked mon.noname-a con 0x7fbb340a9ac0 addr v1:172.21.15.107:6789/0 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 10 monclient(hunting): picked mon.noname-c con 0x7fbb340b4100 addr v1:172.21.15.107:6790/0 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 10 monclient(hunting): picked mon.noname-b con 0x7fbb3413ee90 addr v1:172.21.15.129:6789/0 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3413a650 con 0x7fbb340a9ac0 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb341414c0 con 0x7fbb3413ee90 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb341423d0 con 0x7fbb340b4100 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.742+0000 7fbb3976c640 10 monclient(hunting): authenticate will time out at 1110.010241s 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.744+0000 7fbb32ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbb340a9ac0 0x7fbb341d6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55709/0 (socket says 172.21.15.107:55709) 2024-03-29T11:02:58.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.744+0000 7fbb32ffd640 1 -- 172.21.15.107:0/3589312576 learned_addr learned my addr 172.21.15.107:0/3589312576 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 client.?.objecter ms_handle_connect 0x7fbb340a9ac0 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 client.?.objecter ms_handle_connect 0x7fbb3413ee90 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013624686 0 0) 0x7fbb3413a650 con 0x7fbb340a9ac0 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb1c004eb0 con 0x7fbb340a9ac0 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736023863 0 0) 0x7fbb341414c0 con 0x7fbb3413ee90 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:58.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.746+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb3413a650 con 0x7fbb3413ee90 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.747+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 432112759 0 0) 0x7fbb1c004eb0 con 0x7fbb340a9ac0 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.747+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb341414c0 con 0x7fbb340a9ac0 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.747+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbb24001990 con 0x7fbb340a9ac0 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.747+0000 7fbb2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.747+0000 7fbb2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.747+0000 7fbb2b7fe640 10 monclient(hunting): dump: 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:58.739 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.747+0000 7fbb2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4012026064 0 0) 0x7fbb341414c0 con 0x7fbb340a9ac0 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 >> v1:172.21.15.107:6790/0 conn(0x7fbb340b4100 legacy=0x7fbb341346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 >> v1:172.21.15.129:6789/0 conn(0x7fbb3413ee90 legacy=0x7fbb341de850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:58.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 1 monclient: found mon.a 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb34142560 con 0x7fbb340a9ac0 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.748+0000 7fbb2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb3976c640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb3976c640 10 monclient: _renew_subs 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb3976c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb341d6c20 con 0x7fbb340a9ac0 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb3976c640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:58.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb3976c640 10 monclient: _renew_subs 2024-03-29T11:02:58.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb3976c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:58.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3413def0 con 0x7fbb340a9ac0 2024-03-29T11:02:58.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb3976c640 1 librados: init done 2024-03-29T11:02:58.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb24001af0 con 0x7fbb340a9ac0 2024-03-29T11:02:58.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:58.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.749+0000 7fbb317fa640 4 set_mon_vals no callback set 2024-03-29T11:02:58.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.750+0000 7fbb3976c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:58.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.750+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbb24004c30 con 0x7fbb340a9ac0 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.750+0000 7fbb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.750+0000 7fbb2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.750+0000 7fbb2b7fe640 10 monclient: dump: 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:02:58.743 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.750+0000 7fbb3976c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.751+0000 7fbb3976c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.751+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb10005080 con 0x7fbb340a9ac0 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.752+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fbb2401ac00 con 0x7fbb340a9ac0 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.753+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fbb24084a60 con 0x7fbb340a9ac0 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.753+0000 7fbb2b7fe640 10 client.4419.objecter ms_dispatch 0x7fbb34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.753+0000 7fbb2b7fe640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.753+0000 7fbb2b7fe640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.753+0000 7fbb2b7fe640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.753+0000 7fbb327fc640 10 monclient: get_auth_request con 0x7fbb1c06dba0 auth_method 0 2024-03-29T11:02:58.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.754+0000 7fbb2b7fe640 10 client.4419.objecter ms_handle_connect 0x7fbb1c06dba0 2024-03-29T11:02:58.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.761+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb24050340 con 0x7fbb340a9ac0 2024-03-29T11:02:58.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.761+0000 7fbb2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:58.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:58.761+0000 7fbb2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:59.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.077+0000 7fbb3976c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:59.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.078+0000 7fbb3976c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:59.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.078+0000 7fbb3976c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:02:59.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.078+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb10005330 con 0x7fbb340a9ac0 2024-03-29T11:02:59.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.078+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/3589312576 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fbb24052270 con 0x7fbb340a9ac0 2024-03-29T11:02:59.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.078+0000 7fbb2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:59.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.078+0000 7fbb2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:59.058 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:02:59.059 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.082+0000 7fbb3976c640 10 librados: watch_flush enter 2024-03-29T11:02:59.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.082+0000 7fbb3976c640 10 librados: watch_flush exit 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.082+0000 7fbb3976c640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T11:02:59.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.082+0000 7fbb3976c640 10 client.4419.objecter successfully canceled tick 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.085+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 >> v1:172.21.15.107:6816/882194755 conn(0x7fbb1c06dba0 legacy=0x7fbb1c070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:59.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.085+0000 7fbb3976c640 10 monclient: shutdown 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.085+0000 7fbb3976c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.085+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 >> v1:172.21.15.107:6789/0 conn(0x7fbb340a9ac0 legacy=0x7fbb341d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.085+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 shutdown_connections 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.085+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 >> 172.21.15.107:0/3589312576 conn(0x7fbb340b35e0 msgr2=0x7fbb340b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.085+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 shutdown_connections 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.086+0000 7fbb3976c640 1 -- 172.21.15.107:0/3589312576 wait complete. 2024-03-29T11:02:59.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:02:59.086+0000 7fbb3976c640 1 librados: shutdown 2024-03-29T11:02:59.082 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:05.890 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:06.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.055+0000 7f7109b1c640 10 monclient: get_monmap_and_config 2024-03-29T11:03:06.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.055+0000 7f7109b1c640 10 monclient: build_initial_monmap 2024-03-29T11:03:06.035 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.055+0000 7f7109b1c640 10 monclient: monmap: 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:03:06.056794+0000 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:03:06.056794+0000 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:03:06.036 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:06.038 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.059+0000 7f7109b1c640 1 Processor -- start 2024-03-29T11:03:06.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.059+0000 7f7109b1c640 1 -- start start 2024-03-29T11:03:06.039 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.059+0000 7f7109b1c640 10 monclient: init 2024-03-29T11:03:06.040 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.060+0000 7f7109b1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:06.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.061+0000 7f7109b1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:06.041 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.061+0000 7f7109b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f71040b0860 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:06.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.064+0000 7f7109b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f710413ef30 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:06.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.064+0000 7f71037fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f71040b0860 0x7f710413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44362/0 (socket says 172.21.15.107:44362) 2024-03-29T11:03:06.043 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.064+0000 7f71037fe640 1 -- 172.21.15.107:0/2711673874 learned_addr learned my addr 172.21.15.107:0/2711673874 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:06.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.064+0000 7f7109b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f71040af8e0 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:06.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.064+0000 7f7109b1c640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71040b0c40 con 0x7f710413ef30 2024-03-29T11:03:06.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.064+0000 7f7109b1c640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71040aed10 con 0x7f71040af8e0 2024-03-29T11:03:06.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.064+0000 7f7109b1c640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71040afd50 con 0x7f71040b0860 2024-03-29T11:03:06.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f7109b1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:06.044 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f7109b1c640 10 monclient(hunting): authenticate will time out at 1117.333097s 2024-03-29T11:03:06.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266110177 0 0) 0x7f71040afd50 con 0x7f71040b0860 2024-03-29T11:03:06.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:06.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:06.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:06.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:03:06.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ec003520 con 0x7f71040b0860 2024-03-29T11:03:06.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346478807 0 0) 0x7f71040b0c40 con 0x7f710413ef30 2024-03-29T11:03:06.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:06.045 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:06.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:06.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:03:06.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71040afd50 con 0x7f710413ef30 2024-03-29T11:03:06.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2824642893 0 0) 0x7f70ec003520 con 0x7f71040b0860 2024-03-29T11:03:06.046 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.065+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 >> v1:172.21.15.129:6789/0 conn(0x7f71040af8e0 legacy=0x7f7104149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:06.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.067+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 >> v1:172.21.15.107:6789/0 conn(0x7f710413ef30 legacy=0x7f71040af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:06.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.067+0000 7f71027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:06.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 1 monclient: found mon.noname-b 2024-03-29T11:03:06.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:06.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:03:06.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71040ab1b0 con 0x7f71040b0860 2024-03-29T11:03:06.048 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f70f8001690 con 0x7f71040b0860 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f71027fc640 10 monclient: dump: 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:06.049 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:06.050 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:06.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f7109b1c640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:03:06.050 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.068+0000 7f7109b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:06.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71037fe640 1 -- 172.21.15.107:0/2711673874 >> v1:172.21.15.107:6790/0 conn(0x7f71040b0860 legacy=0x7f710413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:03:06.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71037fe640 1 -- 172.21.15.107:0/2711673874 >> v1:172.21.15.107:6790/0 conn(0x7f71040b0860 legacy=0x7f710413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:03:06.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71037fe640 1 --1- 172.21.15.107:0/2711673874 >> v1:172.21.15.107:6790/0 conn(0x7f71040b0860 0x7f710413e820 :-1 s=OPENED pgs=595948 cs=1 l=1).handle_message read tag failed 2024-03-29T11:03:06.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71037fe640 1 --1- 172.21.15.107:0/2711673874 >> v1:172.21.15.107:6790/0 conn(0x7f71040b0860 0x7f710413e820 :-1 s=OPENED pgs=595948 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:03:06.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.107:6790/0 2024-03-29T11:03:06.052 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 >> v1:172.21.15.107:6790/0 conn(0x7f71040b0860 legacy=0x7f710413e820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 10 monclient(hunting): picked mon.b con 0x7f71040af8e0 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 10 monclient(hunting): picked mon.c con 0x7f70ec005570 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 10 monclient(hunting): picked mon.a con 0x7f710413ef30 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f70ec003520 con 0x7f710413ef30 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f71040afd50 con 0x7f71040af8e0 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f71040aed10 con 0x7f70ec005570 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.070+0000 7f71027fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.107:6790/0 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.072+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190925851 0 0) 0x7f71040aed10 con 0x7f70ec005570 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.072+0000 7f71027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.072+0000 7f71027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.072+0000 7f71027fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:03:06.053 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.072+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.107:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f70ec013000 con 0x7f70ec005570 2024-03-29T11:03:06.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.072+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2098774067 0 0) 0x7f70ec013000 con 0x7f70ec005570 2024-03-29T11:03:06.054 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.072+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 >> v1:172.21.15.129:6789/0 conn(0x7f71040af8e0 legacy=0x7f70ec003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:06.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.073+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 >> v1:172.21.15.107:6789/0 conn(0x7f710413ef30 legacy=0x7f70ec00f750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:06.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:06.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 1 monclient: found mon.c 2024-03-29T11:03:06.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:06.056 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f71040b0c40 con 0x7f70ec005570 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f70fc0020e0 con 0x7f70ec005570 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.074+0000 7f71027fc640 10 monclient: dump: 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:06.057 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.075+0000 7f7109b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.075+0000 7f71027fc640 1 -- 172.21.15.107:0/2711673874 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70fc003970 con 0x7f70ec005570 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.075+0000 7f71027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.075+0000 7f7109b1c640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.075+0000 7f7109b1c640 4 set_mon_vals no callback set 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.076+0000 7f7109b1c640 10 monclient: shutdown 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.076+0000 7f7109b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.076+0000 7f7109b1c640 1 -- 172.21.15.107:0/2711673874 >> v1:172.21.15.107:6790/0 conn(0x7f70ec005570 legacy=0x7f70ec00c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.077+0000 7f7109b1c640 1 -- 172.21.15.107:0/2711673874 shutdown_connections 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.077+0000 7f7109b1c640 1 -- 172.21.15.107:0/2711673874 >> 172.21.15.107:0/2711673874 conn(0x7f710406b660 msgr2=0x7f710406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.077+0000 7f7109b1c640 1 -- 172.21.15.107:0/2711673874 shutdown_connections 2024-03-29T11:03:06.058 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.077+0000 7f7109b1c640 1 -- 172.21.15.107:0/2711673874 wait complete. 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.078+0000 7f7109b1c640 10 monclient: build_initial_monmap 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.078+0000 7f7109b1c640 10 monclient: monmap: 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:03:06.061 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.078+0000 7f7109b1c640 1 librados: starting msgr at 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.078+0000 7f7109b1c640 1 librados: starting objecter 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.078+0000 7f7109b1c640 1 Processor -- start 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.079+0000 7f7109b1c640 1 -- start start 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.079+0000 7f7109b1c640 1 librados: setting wanted keys 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.079+0000 7f7109b1c640 1 librados: calling monclient init 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.079+0000 7f7109b1c640 10 monclient: init 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.079+0000 7f7109b1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.079+0000 7f7109b1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.080+0000 7f7109b1c640 10 monclient(hunting): picked mon.noname-c con 0x7f70ec005570 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.080+0000 7f7109b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f71040af8e0 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.080+0000 7f7109b1c640 10 monclient(hunting): picked mon.noname-a con 0x7f710413ef30 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.080+0000 7f7109b1c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7104133150 con 0x7f710413ef30 2024-03-29T11:03:06.062 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.080+0000 7f7109b1c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71041d6070 con 0x7f71040af8e0 2024-03-29T11:03:06.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.080+0000 7f7109b1c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71041d6f40 con 0x7f70ec005570 2024-03-29T11:03:06.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.080+0000 7f7109b1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:06.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.080+0000 7f7109b1c640 10 monclient(hunting): authenticate will time out at 1117.348067s 2024-03-29T11:03:06.063 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.082+0000 7f71037fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f70ec005570 0x7f71041d6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47300/0 (socket says 172.21.15.107:47300) 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.082+0000 7f71037fe640 1 -- 172.21.15.107:0/1970410968 learned_addr learned my addr 172.21.15.107:0/1970410968 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.082+0000 7f70ebfff640 10 client.?.objecter ms_handle_connect 0x7f70ec005570 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.082+0000 7f70ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630441533 0 0) 0x7f71041d6f40 con 0x7f70ec005570 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f0004eb0 con 0x7f70ec005570 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3505525626 0 0) 0x7f70f0004eb0 con 0x7f70ec005570 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71041d6f40 con 0x7f70ec005570 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f70f8001b10 con 0x7f70ec005570 2024-03-29T11:03:06.064 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 10 monclient(hunting): dump: 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.083+0000 7f70ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1013765717 0 0) 0x7f71041d6f40 con 0x7f70ec005570 2024-03-29T11:03:06.065 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 >> v1:172.21.15.129:6789/0 conn(0x7f71040af8e0 legacy=0x7f7104132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:06.066 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 >> v1:172.21.15.107:6789/0 conn(0x7f710413ef30 legacy=0x7f7104132a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 1 monclient: found mon.c 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71040719f0 con 0x7f70ec005570 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 10 monclient: _finish_auth 0 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f7109b1c640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f7109b1c640 10 monclient: _renew_subs 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f7109b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7104071be0 con 0x7f70ec005570 2024-03-29T11:03:06.067 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f7109b1c640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f7109b1c640 10 monclient: _renew_subs 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f7109b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f710413ae50 con 0x7f70ec005570 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f7109b1c640 1 librados: init done 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70f8001250 con 0x7f70ec005570 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f70ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.084+0000 7f71017fa640 4 set_mon_vals no callback set 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.086+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f70f8001b10 con 0x7f70ec005570 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.086+0000 7f70ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.086+0000 7f70ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.086+0000 7f70ebfff640 10 monclient: dump: 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:06.068 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:06.069 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:06.069 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:06.069 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:06.069 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:06.069 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:06.069 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:06.069 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.087+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f70f801a900 con 0x7f70ec005570 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f70f8084580 con 0x7f70ec005570 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f70ebfff640 10 client.4466.objecter ms_dispatch 0x7f7104000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f70ebfff640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f70ebfff640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f70ebfff640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f7109b1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f7109b1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f7109b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e4005080 con 0x7f70ec005570 2024-03-29T11:03:06.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f7102ffd640 10 monclient: get_auth_request con 0x7f70f006df70 auth_method 0 2024-03-29T11:03:06.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.088+0000 7f70ebfff640 10 client.4466.objecter ms_handle_connect 0x7f70f006df70 2024-03-29T11:03:06.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.096+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70f804fec0 con 0x7f70ec005570 2024-03-29T11:03:06.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.096+0000 7f70ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:06.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.096+0000 7f70ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:06.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.418+0000 7f7109b1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:06.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.418+0000 7f7109b1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:06.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.418+0000 7f7109b1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:06.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.418+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70e4005330 con 0x7f70ec005570 2024-03-29T11:03:06.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.419+0000 7f70ebfff640 1 -- 172.21.15.107:0/1970410968 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f70f8083c30 con 0x7f70ec005570 2024-03-29T11:03:06.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.419+0000 7f70ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:06.399 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.419+0000 7f70ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:06.400 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:03:06.400 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:06.401 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.422+0000 7f7109b1c640 10 librados: watch_flush enter 2024-03-29T11:03:06.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.422+0000 7f7109b1c640 10 librados: watch_flush exit 2024-03-29T11:03:06.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.422+0000 7f7109b1c640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T11:03:06.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.422+0000 7f7109b1c640 10 client.4466.objecter successfully canceled tick 2024-03-29T11:03:06.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.425+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 >> v1:172.21.15.107:6816/882194755 conn(0x7f70f006df70 legacy=0x7f70f0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:06.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.425+0000 7f7109b1c640 10 monclient: shutdown 2024-03-29T11:03:06.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.425+0000 7f7109b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:06.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.425+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 >> v1:172.21.15.107:6790/0 conn(0x7f70ec005570 legacy=0x7f71041d6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:06.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.425+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 shutdown_connections 2024-03-29T11:03:06.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.425+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 >> 172.21.15.107:0/1970410968 conn(0x7f710406b660 msgr2=0x7f7104134810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:06.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.425+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 shutdown_connections 2024-03-29T11:03:06.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.425+0000 7f7109b1c640 1 -- 172.21.15.107:0/1970410968 wait complete. 2024-03-29T11:03:06.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:06.426+0000 7f7109b1c640 1 librados: shutdown 2024-03-29T11:03:06.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:13.229 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:13.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.400+0000 7f97a1e0c640 10 monclient: get_monmap_and_config 2024-03-29T11:03:13.380 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.400+0000 7f97a1e0c640 10 monclient: build_initial_monmap 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.401+0000 7f97a1e0c640 10 monclient: monmap: 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:03:13.401946+0000 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:03:13.401946+0000 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:03:13.381 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:13.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.403+0000 7f97a1e0c640 1 Processor -- start 2024-03-29T11:03:13.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.403+0000 7f97a1e0c640 1 -- start start 2024-03-29T11:03:13.382 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.403+0000 7f97a1e0c640 10 monclient: init 2024-03-29T11:03:13.383 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:13.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:03:13.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 10 monclient(hunting): picked mon.noname-a con 0x7f979c0b2800 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:13.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 10 monclient(hunting): picked mon.noname-b con 0x7f979c0b3220 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:13.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 10 monclient(hunting): picked mon.noname-c con 0x7f979c1439c0 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:13.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c0b3840 con 0x7f979c0b2800 2024-03-29T11:03:13.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c06f230 con 0x7f979c1439c0 2024-03-29T11:03:13.385 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c06ef50 con 0x7f979c0b3220 2024-03-29T11:03:13.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:13.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.404+0000 7f97a1e0c640 10 monclient(hunting): authenticate will time out at 1124.672267s 2024-03-29T11:03:13.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.405+0000 7f979bfff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f979c0b3220 0x7f979c146710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41260/0 (socket says 172.21.15.107:41260) 2024-03-29T11:03:13.386 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.405+0000 7f979bfff640 1 -- 172.21.15.107:0/1975907395 learned_addr learned my addr 172.21.15.107:0/1975907395 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:13.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.407+0000 7f979b7fe640 1 -- 172.21.15.107:0/1975907395 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256865397 0 0) 0x7f979c0b3840 con 0x7f979c0b2800 2024-03-29T11:03:13.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.407+0000 7f979b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:03:13.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.407+0000 7f979b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:13.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.407+0000 7f979b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:13.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.407+0000 7f979b7fe640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:03:13.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.407+0000 7f979b7fe640 1 -- 172.21.15.107:0/1975907395 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9784003520 con 0x7f979c0b2800 2024-03-29T11:03:13.387 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 1 -- 172.21.15.107:0/1975907395 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2409707643 0 0) 0x7f9784003520 con 0x7f979c0b2800 2024-03-29T11:03:13.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 1 -- 172.21.15.107:0/1975907395 >> v1:172.21.15.107:6790/0 conn(0x7f979c0b3220 legacy=0x7f979c146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:03:13.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 1 -- 172.21.15.107:0/1975907395 >> v1:172.21.15.129:6789/0 conn(0x7f979c1439c0 legacy=0x7f979c151c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:13.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:03:13.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:03:13.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:13.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:03:13.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 1 -- 172.21.15.107:0/1975907395 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c071d30 con 0x7f979c0b2800 2024-03-29T11:03:13.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:03:13.388 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 1 -- 172.21.15.107:0/1975907395 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f978c001690 con 0x7f979c0b2800 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.408+0000 7f979b7fe640 10 monclient: dump: 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:13.389 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:13.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f97a1e0c640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:03:13.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f97a1e0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:13.390 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f979b7fe640 1 -- 172.21.15.107:0/1975907395 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f978c001860 con 0x7f979c0b2800 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f979b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f979b7fe640 1 -- 172.21.15.107:0/1975907395 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f978c001cb0 con 0x7f979c0b2800 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f979b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f979b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f979b7fe640 10 monclient: dump: 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:13.391 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f97a1e0c640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.409+0000 7f97a1e0c640 4 set_mon_vals no callback set 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.410+0000 7f97a1e0c640 10 monclient: shutdown 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.410+0000 7f97a1e0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.410+0000 7f97a1e0c640 1 -- 172.21.15.107:0/1975907395 >> v1:172.21.15.107:6789/0 conn(0x7f979c0b2800 legacy=0x7f979c1432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.411+0000 7f97a1e0c640 1 -- 172.21.15.107:0/1975907395 shutdown_connections 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.411+0000 7f97a1e0c640 1 -- 172.21.15.107:0/1975907395 >> 172.21.15.107:0/1975907395 conn(0x7f979c066a00 msgr2=0x7f979c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.411+0000 7f97a1e0c640 1 -- 172.21.15.107:0/1975907395 shutdown_connections 2024-03-29T11:03:13.392 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.411+0000 7f97a1e0c640 1 -- 172.21.15.107:0/1975907395 wait complete. 2024-03-29T11:03:13.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.412+0000 7f97a1e0c640 10 monclient: build_initial_monmap 2024-03-29T11:03:13.393 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.412+0000 7f97a1e0c640 10 monclient: monmap: 2024-03-29T11:03:13.393 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:13.393 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:13.393 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:03:13.393 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:03:13.393 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:13.393 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.413+0000 7f97a1e0c640 1 librados: starting msgr at 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.413+0000 7f97a1e0c640 1 librados: starting objecter 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.413+0000 7f97a1e0c640 1 Processor -- start 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.413+0000 7f97a1e0c640 1 -- start start 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.413+0000 7f97a1e0c640 1 librados: setting wanted keys 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.413+0000 7f97a1e0c640 1 librados: calling monclient init 2024-03-29T11:03:13.394 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.413+0000 7f97a1e0c640 10 monclient: init 2024-03-29T11:03:13.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:13.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:13.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 10 monclient(hunting): picked mon.noname-b con 0x7f979c0b2800 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:13.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 10 monclient(hunting): picked mon.noname-a con 0x7f979c0b3220 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:13.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 10 monclient(hunting): picked mon.noname-c con 0x7f979c1439c0 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:13.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c1215a0 con 0x7f979c0b3220 2024-03-29T11:03:13.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c1db670 con 0x7f979c0b2800 2024-03-29T11:03:13.396 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c120360 con 0x7f979c1439c0 2024-03-29T11:03:13.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:13.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.414+0000 7f97a1e0c640 10 monclient(hunting): authenticate will time out at 1124.682459s 2024-03-29T11:03:13.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97a160b640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f979c1439c0 0x7f979c13df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42340/0 (socket says 172.21.15.107:42340) 2024-03-29T11:03:13.397 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97a160b640 1 -- 172.21.15.107:0/2471520370 learned_addr learned my addr 172.21.15.107:0/2471520370 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97997fa640 10 client.?.objecter ms_handle_connect 0x7f979c1439c0 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501610609 0 0) 0x7f979c120360 con 0x7f979c1439c0 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97997fa640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.417+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9788004eb0 con 0x7f979c1439c0 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.418+0000 7f97997fa640 10 client.?.objecter ms_handle_connect 0x7f979c0b3220 2024-03-29T11:03:13.398 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.419+0000 7f97997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.419+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621638272 0 0) 0x7f979c1215a0 con 0x7f979c0b3220 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c120360 con 0x7f979c0b3220 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3966808843 0 0) 0x7f9788004eb0 con 0x7f979c1439c0 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f979c1215a0 con 0x7f979c1439c0 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9794001a50 con 0x7f979c1439c0 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:13.402 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 10 monclient(hunting): dump: 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.420+0000 7f97997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 682513835 0 0) 0x7f979c120360 con 0x7f979c0b3220 2024-03-29T11:03:13.403 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9788004eb0 con 0x7f979c0b3220 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9790001510 con 0x7f979c0b3220 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 10 monclient(hunting): dump: 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:13.404 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3157930272 0 0) 0x7f9788004eb0 con 0x7f979c0b3220 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 >> v1:172.21.15.107:6790/0 conn(0x7f979c1439c0 legacy=0x7f979c13df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 >> v1:172.21.15.129:6789/0 conn(0x7f979c0b2800 legacy=0x7f979c11fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 1 monclient: found mon.a 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c147720 con 0x7f979c0b3220 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 10 monclient: _finish_auth 0 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.421+0000 7f97997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.422+0000 7f97a1e0c640 5 monclient: authenticate success, global_id 4428 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.422+0000 7f97a1e0c640 10 monclient: _renew_subs 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.422+0000 7f97a1e0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.422+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c1dc2a0 con 0x7f979c0b3220 2024-03-29T11:03:13.405 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.422+0000 7f97a1e0c640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:13.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.422+0000 7f97a1e0c640 10 monclient: _renew_subs 2024-03-29T11:03:13.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.422+0000 7f97a1e0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:13.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.422+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c14a3c0 con 0x7f979c0b3220 2024-03-29T11:03:13.406 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.422+0000 7f97a1e0c640 1 librados: init done 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.424+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9790001670 con 0x7f979c0b3220 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.424+0000 7f97997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.424+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9790001d60 con 0x7f979c0b3220 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.424+0000 7f97997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.424+0000 7f97997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.424+0000 7f97997fa640 10 monclient: dump: 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:13.408 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.425+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f979001aa00 con 0x7f979c0b3220 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.425+0000 7f979affd640 4 set_mon_vals no callback set 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.426+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f9790084a00 con 0x7f979c0b3220 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.426+0000 7f97997fa640 10 client.4428.objecter ms_dispatch 0x7f979c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.426+0000 7f97997fa640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.426+0000 7f97997fa640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.426+0000 7f97a0e0a640 10 monclient: get_auth_request con 0x7f978806df10 auth_method 0 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.426+0000 7f97a1e0c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.426+0000 7f97a1e0c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.426+0000 7f97a1e0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.426+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c005080 con 0x7f979c0b3220 2024-03-29T11:03:13.409 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.427+0000 7f97997fa640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-29T11:03:13.410 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.429+0000 7f97997fa640 10 client.4428.objecter ms_handle_connect 0x7f978806df10 2024-03-29T11:03:13.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.435+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9790050110 con 0x7f979c0b3220 2024-03-29T11:03:13.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.435+0000 7f97997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:13.415 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.435+0000 7f97997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:13.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.758+0000 7f97a1e0c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:13.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.758+0000 7f97a1e0c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:13.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.758+0000 7f97a1e0c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:13.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.758+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f977c005330 con 0x7f979c0b3220 2024-03-29T11:03:13.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.760+0000 7f97997fa640 1 -- 172.21.15.107:0/2471520370 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f9790052210 con 0x7f979c0b3220 2024-03-29T11:03:13.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.760+0000 7f97997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:13.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.760+0000 7f97997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:13.740 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:03:13.740 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:13.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.765+0000 7f97a1e0c640 10 librados: watch_flush enter 2024-03-29T11:03:13.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.765+0000 7f97a1e0c640 10 librados: watch_flush exit 2024-03-29T11:03:13.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.765+0000 7f97a1e0c640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-29T11:03:13.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.765+0000 7f97a1e0c640 10 client.4428.objecter successfully canceled tick 2024-03-29T11:03:13.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.765+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 >> v1:172.21.15.107:6816/882194755 conn(0x7f978806df10 legacy=0x7f97880703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:13.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.765+0000 7f97a1e0c640 10 monclient: shutdown 2024-03-29T11:03:13.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.765+0000 7f97a1e0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:13.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.765+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 >> v1:172.21.15.107:6789/0 conn(0x7f979c0b3220 legacy=0x7f979c1dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:13.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.766+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 shutdown_connections 2024-03-29T11:03:13.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.766+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 >> 172.21.15.107:0/2471520370 conn(0x7f979c066a00 msgr2=0x7f979c06e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:13.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.766+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 shutdown_connections 2024-03-29T11:03:13.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.766+0000 7f97a1e0c640 1 -- 172.21.15.107:0/2471520370 wait complete. 2024-03-29T11:03:13.746 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:13.766+0000 7f97a1e0c640 1 librados: shutdown 2024-03-29T11:03:13.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:20.569 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:20.713 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.733+0000 7f058d762640 10 monclient: get_monmap_and_config 2024-03-29T11:03:20.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.733+0000 7f058d762640 10 monclient: build_initial_monmap 2024-03-29T11:03:20.714 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.733+0000 7f058d762640 10 monclient: monmap: 2024-03-29T11:03:20.714 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:20.714 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:20.714 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:03:20.734306+0000 2024-03-29T11:03:20.714 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:03:20.734306+0000 2024-03-29T11:03:20.714 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:20.714 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:20.715 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:20.715 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:03:20.715 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:03:20.715 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.734+0000 7f058d762640 1 Processor -- start 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 1 -- start start 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 10 monclient: init 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 10 monclient(hunting): picked mon.noname-c con 0x7f05880ad1a0 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 10 monclient(hunting): picked mon.noname-b con 0x7f05880ae710 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 10 monclient(hunting): picked mon.noname-a con 0x7f05881404a0 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880aed30 con 0x7f05881404a0 2024-03-29T11:03:20.717 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880afe20 con 0x7f05880ad1a0 2024-03-29T11:03:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058813d8f0 con 0x7f05880ae710 2024-03-29T11:03:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.735+0000 7f058d762640 10 monclient(hunting): authenticate will time out at 1132.003862s 2024-03-29T11:03:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.736+0000 7f057ffff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f05880ae710 0x7f058813cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33574/0 (socket says 172.21.15.107:33574) 2024-03-29T11:03:20.718 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.736+0000 7f057ffff640 1 -- 172.21.15.107:0/1294736974 learned_addr learned my addr 172.21.15.107:0/1294736974 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.737+0000 7f05877fe640 1 -- 172.21.15.107:0/1294736974 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561143074 0 0) 0x7f058813d8f0 con 0x7f05880ae710 2024-03-29T11:03:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.737+0000 7f05877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:03:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.737+0000 7f05877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.737+0000 7f05877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.737+0000 7f05877fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:03:20.719 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.737+0000 7f05877fe640 1 -- 172.21.15.107:0/1294736974 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0574003520 con 0x7f05880ae710 2024-03-29T11:03:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 1 -- 172.21.15.107:0/1294736974 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012871609 0 0) 0x7f0574003520 con 0x7f05880ae710 2024-03-29T11:03:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 1 -- 172.21.15.107:0/1294736974 >> v1:172.21.15.129:6789/0 conn(0x7f05880ad1a0 legacy=0x7f058813fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 1 -- 172.21.15.107:0/1294736974 >> v1:172.21.15.107:6789/0 conn(0x7f05881404a0 legacy=0x7f058813d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:03:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 10 monclient: _finish_hunting 0 2024-03-29T11:03:20.722 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 1 monclient: found mon.noname-b 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 1 -- 172.21.15.107:0/1294736974 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880af1c0 con 0x7f05880ae710 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 10 monclient: _finish_auth 0 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 10 monclient: _check_auth_tickets 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 1 -- 172.21.15.107:0/1294736974 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f0570001690 con 0x7f05880ae710 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 10 monclient: dump: 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:20.723 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 1 -- 172.21.15.107:0/1294736974 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0570001920 con 0x7f05880ae710 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.741+0000 7f05877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.742+0000 7f05877fe640 1 -- 172.21.15.107:0/1294736974 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f0570001cb0 con 0x7f05880ae710 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.742+0000 7f05877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:20.724 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.742+0000 7f05877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.742+0000 7f05877fe640 10 monclient: dump: 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.742+0000 7f058d762640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.742+0000 7f058d762640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:20.725 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.742+0000 7f058d762640 4 set_mon_vals no callback set 2024-03-29T11:03:20.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.743+0000 7f058d762640 10 monclient: shutdown 2024-03-29T11:03:20.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.743+0000 7f058d762640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:20.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.743+0000 7f058d762640 1 -- 172.21.15.107:0/1294736974 >> v1:172.21.15.107:6790/0 conn(0x7f05880ae710 legacy=0x7f058813cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:20.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.743+0000 7f058d762640 1 -- 172.21.15.107:0/1294736974 shutdown_connections 2024-03-29T11:03:20.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.743+0000 7f058d762640 1 -- 172.21.15.107:0/1294736974 >> 172.21.15.107:0/1294736974 conn(0x7f05880673e0 msgr2=0x7f05880b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:20.727 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.744+0000 7f058d762640 1 -- 172.21.15.107:0/1294736974 shutdown_connections 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.744+0000 7f058d762640 1 -- 172.21.15.107:0/1294736974 wait complete. 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.745+0000 7f058d762640 10 monclient: build_initial_monmap 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.745+0000 7f058d762640 10 monclient: monmap: 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:03:20.728 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.745+0000 7f058d762640 1 librados: starting msgr at 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.745+0000 7f058d762640 1 librados: starting objecter 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.745+0000 7f058d762640 1 Processor -- start 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.745+0000 7f058d762640 1 -- start start 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.745+0000 7f058d762640 1 librados: setting wanted keys 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.745+0000 7f058d762640 1 librados: calling monclient init 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.745+0000 7f058d762640 10 monclient: init 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 10 monclient(hunting): picked mon.noname-b con 0x7f05880ad1a0 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 10 monclient(hunting): picked mon.noname-a con 0x7f05880ae710 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 10 monclient(hunting): picked mon.noname-c con 0x7f05881404a0 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880ac2f0 con 0x7f05880ae710 2024-03-29T11:03:20.729 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05881328b0 con 0x7f05880ad1a0 2024-03-29T11:03:20.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0588132b90 con 0x7f05881404a0 2024-03-29T11:03:20.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:20.730 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.746+0000 7f058d762640 10 monclient(hunting): authenticate will time out at 1132.014493s 2024-03-29T11:03:20.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.749+0000 7f058cf61640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f05881404a0 0x7f05881336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38316/0 (socket says 172.21.15.107:38316) 2024-03-29T11:03:20.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.749+0000 7f058cf61640 1 -- 172.21.15.107:0/135318370 learned_addr learned my addr 172.21.15.107:0/135318370 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:20.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.750+0000 7f05857fa640 10 client.?.objecter ms_handle_connect 0x7f05881404a0 2024-03-29T11:03:20.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.750+0000 7f05857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:20.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.750+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372943971 0 0) 0x7f0588132b90 con 0x7f05881404a0 2024-03-29T11:03:20.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.750+0000 7f05857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:20.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.750+0000 7f05857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:20.735 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.750+0000 7f05857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.750+0000 7f05857fa640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.750+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0568004eb0 con 0x7f05881404a0 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 10 client.?.objecter ms_handle_connect 0x7f05880ad1a0 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 10 client.?.objecter ms_handle_connect 0x7f05880ae710 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429823355 0 0) 0x7f05881328b0 con 0x7f05880ad1a0 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:03:20.736 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0588132b90 con 0x7f05880ad1a0 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433390495 0 0) 0x7f05880ac2f0 con 0x7f05880ae710 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.751+0000 7f05857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05881328b0 con 0x7f05880ae710 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1056846029 0 0) 0x7f0568004eb0 con 0x7f05881404a0 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05880ac2f0 con 0x7f05881404a0 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 478545410 0 0) 0x7f0588132b90 con 0x7f05880ad1a0 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0568004eb0 con 0x7f05880ad1a0 2024-03-29T11:03:20.737 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f0580001a50 con 0x7f05881404a0 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): dump: 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:20.738 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f0578001830 con 0x7f05880ad1a0 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 10 monclient(hunting): dump: 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:20.739 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2547803586 0 0) 0x7f05881328b0 con 0x7f05880ae710 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0588132b90 con 0x7f05880ae710 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.752+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f05700013b0 con 0x7f05880ae710 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 10 monclient(hunting): dump: 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:20.740 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 938043499 0 0) 0x7f0568004eb0 con 0x7f05880ad1a0 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 >> v1:172.21.15.107:6790/0 conn(0x7f05881404a0 legacy=0x7f05881336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 >> v1:172.21.15.107:6789/0 conn(0x7f05880ae710 legacy=0x7f0588132fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 10 monclient: _finish_hunting 0 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 1 monclient: found mon.b 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0588131a20 con 0x7f05880ad1a0 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 10 monclient: _finish_auth 0 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 10 monclient: _check_auth_tickets 2024-03-29T11:03:20.741 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.753+0000 7f05857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.754+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05780035f0 con 0x7f05880ad1a0 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.754+0000 7f05857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.754+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f0578003900 con 0x7f05880ad1a0 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.754+0000 7f05857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.754+0000 7f05857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.754+0000 7f05857fa640 10 monclient: dump: 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:20.742 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.754+0000 7f0586ffd640 4 set_mon_vals no callback set 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.755+0000 7f058d762640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.755+0000 7f058d762640 10 monclient: _renew_subs 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.755+0000 7f058d762640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.755+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058813b450 con 0x7f05880ad1a0 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.755+0000 7f058d762640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.755+0000 7f058d762640 10 monclient: _renew_subs 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.755+0000 7f058d762640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.755+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05880ac780 con 0x7f05880ad1a0 2024-03-29T11:03:20.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.755+0000 7f058d762640 1 librados: init done 2024-03-29T11:03:20.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.757+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f05780198d0 con 0x7f05880ad1a0 2024-03-29T11:03:20.744 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.757+0000 7f058d762640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.757+0000 7f058d762640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.757+0000 7f058d762640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.757+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058814c810 con 0x7f05880ad1a0 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.758+0000 7f057ffff640 10 monclient: get_auth_request con 0x7f056806e300 auth_method 0 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.758+0000 7f05857fa640 10 client.4405.objecter ms_handle_connect 0x7f056806e300 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.758+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f0578083700 con 0x7f05880ad1a0 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.758+0000 7f05857fa640 10 client.4405.objecter ms_dispatch 0x7f0588000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.758+0000 7f05857fa640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.758+0000 7f05857fa640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:20.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.759+0000 7f05857fa640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T11:03:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.767+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f058814c810 con 0x7f05880ad1a0 2024-03-29T11:03:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.767+0000 7f05857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:20.747 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:20.768+0000 7f05857fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.094+0000 7f058d762640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.094+0000 7f058d762640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.094+0000 7f058d762640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:21.074 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.094+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f058814c810 con 0x7f05880ad1a0 2024-03-29T11:03:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.097+0000 7f05857fa640 1 -- 172.21.15.107:0/135318370 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f058814c810 con 0x7f05880ad1a0 2024-03-29T11:03:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.097+0000 7f05857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:21.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.097+0000 7f05857fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:21.077 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:03:21.077 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.099+0000 7f058d762640 10 librados: watch_flush enter 2024-03-29T11:03:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.100+0000 7f058d762640 10 librados: watch_flush exit 2024-03-29T11:03:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.100+0000 7f058d762640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T11:03:21.079 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.100+0000 7f058d762640 10 client.4405.objecter successfully canceled tick 2024-03-29T11:03:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.100+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 >> v1:172.21.15.107:6816/882194755 conn(0x7f056806e300 legacy=0x7f05680707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.100+0000 7f058d762640 10 monclient: shutdown 2024-03-29T11:03:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.100+0000 7f058d762640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.100+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 >> v1:172.21.15.129:6789/0 conn(0x7f05880ad1a0 legacy=0x7f05880aa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:21.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.100+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 shutdown_connections 2024-03-29T11:03:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.100+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 >> 172.21.15.107:0/135318370 conn(0x7f05880673e0 msgr2=0x7f05880b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.101+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 shutdown_connections 2024-03-29T11:03:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.101+0000 7f058d762640 1 -- 172.21.15.107:0/135318370 wait complete. 2024-03-29T11:03:21.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:21.102+0000 7f058d762640 1 librados: shutdown 2024-03-29T11:03:21.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:27.906 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:28.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.090+0000 7f7a5622b640 10 monclient: get_monmap_and_config 2024-03-29T11:03:28.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.090+0000 7f7a5622b640 10 monclient: build_initial_monmap 2024-03-29T11:03:28.070 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.090+0000 7f7a5622b640 10 monclient: monmap: 2024-03-29T11:03:28.070 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:28.070 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:28.070 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:03:28.091714+0000 2024-03-29T11:03:28.071 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:03:28.091714+0000 2024-03-29T11:03:28.071 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:28.071 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:28.071 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:28.071 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:03:28.071 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:03:28.071 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:28.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.092+0000 7f7a5622b640 1 Processor -- start 2024-03-29T11:03:28.071 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.092+0000 7f7a5622b640 1 -- start start 2024-03-29T11:03:28.072 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.092+0000 7f7a5622b640 10 monclient: init 2024-03-29T11:03:28.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.093+0000 7f7a5622b640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:28.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.093+0000 7f7a5622b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:28.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.093+0000 7f7a5622b640 10 monclient(hunting): picked mon.noname-b con 0x7f7a480389e0 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:28.075 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.093+0000 7f7a5622b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a48038dc0 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:28.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.093+0000 7f7a5622b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a48030660 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:28.076 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.097+0000 7f7a5622b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a480c54c0 con 0x7f7a48038dc0 2024-03-29T11:03:28.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.097+0000 7f7a5622b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a48036430 con 0x7f7a48030660 2024-03-29T11:03:28.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.097+0000 7f7a5622b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a48039840 con 0x7f7a480389e0 2024-03-29T11:03:28.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.097+0000 7f7a5622b640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:28.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.097+0000 7f7a5622b640 10 monclient(hunting): authenticate will time out at 1139.365165s 2024-03-29T11:03:28.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.097+0000 7f7a55a2a640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f7a48030660 0x7f7a480d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:33795/0 (socket says 172.21.15.107:33795) 2024-03-29T11:03:28.077 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.097+0000 7f7a55a2a640 1 -- 172.21.15.107:0/2649786088 learned_addr learned my addr 172.21.15.107:0/2649786088 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.100+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737545970 0 0) 0x7f7a48036430 con 0x7f7a48030660 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.100+0000 7f7a47fff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.100+0000 7f7a47fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.100+0000 7f7a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.100+0000 7f7a47fff640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.100+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a3c003520 con 0x7f7a48030660 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202618862 0 0) 0x7f7a48039840 con 0x7f7a480389e0 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:28.080 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:28.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:03:28.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a48036430 con 0x7f7a480389e0 2024-03-29T11:03:28.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1858094937 0 0) 0x7f7a3c003520 con 0x7f7a48030660 2024-03-29T11:03:28.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 >> v1:172.21.15.107:6790/0 conn(0x7f7a480389e0 legacy=0x7f7a48033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:28.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 >> v1:172.21.15.107:6789/0 conn(0x7f7a48038dc0 legacy=0x7f7a480cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:28.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient: _finish_hunting 0 2024-03-29T11:03:28.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 1 monclient: found mon.noname-c 2024-03-29T11:03:28.081 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48035690 con 0x7f7a48030660 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient: _finish_auth 0 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient: _check_auth_tickets 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7a5004f730 con 0x7f7a48030660 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a47fff640 10 monclient: dump: 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:28.082 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a5622b640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:03:28.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.101+0000 7f7a5622b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.102+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a500008d0 con 0x7f7a48030660 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.102+0000 7f7a47fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.102+0000 7f7a5622b640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.102+0000 7f7a5622b640 4 set_mon_vals no callback set 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.103+0000 7f7a5622b640 10 monclient: shutdown 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.103+0000 7f7a5622b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.103+0000 7f7a5622b640 1 -- 172.21.15.107:0/2649786088 >> v1:172.21.15.129:6789/0 conn(0x7f7a48030660 legacy=0x7f7a480d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.103+0000 7f7a47fff640 1 -- 172.21.15.107:0/2649786088 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7a500758b0 con 0x7f7a48030660 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.103+0000 7f7a47fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:03:28.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.104+0000 7f7a5622b640 1 -- 172.21.15.107:0/2649786088 shutdown_connections 2024-03-29T11:03:28.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.104+0000 7f7a5622b640 1 -- 172.21.15.107:0/2649786088 >> 172.21.15.107:0/2649786088 conn(0x7f7a48005c00 msgr2=0x7f7a48006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:28.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.104+0000 7f7a5622b640 1 -- 172.21.15.107:0/2649786088 shutdown_connections 2024-03-29T11:03:28.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.105+0000 7f7a5622b640 1 -- 172.21.15.107:0/2649786088 wait complete. 2024-03-29T11:03:28.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.106+0000 7f7a5622b640 10 monclient: build_initial_monmap 2024-03-29T11:03:28.086 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.107+0000 7f7a5622b640 10 monclient: monmap: 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.107+0000 7f7a5622b640 1 librados: starting msgr at 2024-03-29T11:03:28.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.107+0000 7f7a5622b640 1 librados: starting objecter 2024-03-29T11:03:28.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.107+0000 7f7a5622b640 1 Processor -- start 2024-03-29T11:03:28.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.108+0000 7f7a5622b640 1 -- start start 2024-03-29T11:03:28.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.108+0000 7f7a5622b640 1 librados: setting wanted keys 2024-03-29T11:03:28.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.108+0000 7f7a5622b640 1 librados: calling monclient init 2024-03-29T11:03:28.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.108+0000 7f7a5622b640 10 monclient: init 2024-03-29T11:03:28.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.109+0000 7f7a5622b640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:28.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.109+0000 7f7a5622b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:28.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.113+0000 7f7a5622b640 10 monclient(hunting): picked mon.noname-b con 0x7f7a48030660 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:28.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.113+0000 7f7a55229640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f7a48030660 0x7f7a4802d630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:33917/0 (socket says 172.21.15.107:33917) 2024-03-29T11:03:28.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.113+0000 7f7a55229640 1 -- 172.21.15.107:0/3086099018 learned_addr learned my addr 172.21.15.107:0/3086099018 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:28.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.113+0000 7f7a45ffb640 10 client.?.objecter ms_handle_connect 0x7f7a48030660 2024-03-29T11:03:28.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a45ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:28.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a5622b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a480389e0 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:28.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a5622b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a480c8150 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:28.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4802eba0 con 0x7f7a480389e0 2024-03-29T11:03:28.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4802ee80 con 0x7f7a48030660 2024-03-29T11:03:28.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4802f2a0 con 0x7f7a480c8150 2024-03-29T11:03:28.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a5622b640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:28.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a5622b640 10 monclient(hunting): authenticate will time out at 1139.383141s 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462266337 0 0) 0x7f7a4802ee80 con 0x7f7a48030660 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a45ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a45ffb640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.115+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a34004eb0 con 0x7f7a48030660 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 client.?.objecter ms_handle_connect 0x7f7a480c8150 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636203223 0 0) 0x7f7a34004eb0 con 0x7f7a48030660 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a4802ee80 con 0x7f7a48030660 2024-03-29T11:03:28.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7a4c001a30 con 0x7f7a48030660 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 monclient(hunting): dump: 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:28.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492413053 0 0) 0x7f7a4802f2a0 con 0x7f7a480c8150 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.116+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a34004eb0 con 0x7f7a480c8150 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 10 client.?.objecter ms_handle_connect 0x7f7a480389e0 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1745958093 0 0) 0x7f7a34004eb0 con 0x7f7a480c8150 2024-03-29T11:03:28.099 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a4802f2a0 con 0x7f7a480c8150 2024-03-29T11:03:28.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1337523954 0 0) 0x7f7a4802ee80 con 0x7f7a48030660 2024-03-29T11:03:28.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 >> v1:172.21.15.107:6790/0 conn(0x7f7a480c8150 legacy=0x7f7a481673c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:28.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 >> v1:172.21.15.107:6789/0 conn(0x7f7a480389e0 legacy=0x7f7a4802f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:28.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:03:28.100 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 1 monclient: found mon.b 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4802f430 con 0x7f7a48030660 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 10 monclient: _finish_auth 0 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a5622b640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a5622b640 10 monclient: _renew_subs 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.117+0000 7f7a5622b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4802ffd0 con 0x7f7a48030660 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a5622b640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a5622b640 10 monclient: _renew_subs 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a5622b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:28.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a48168cc0 con 0x7f7a48030660 2024-03-29T11:03:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a5622b640 1 librados: init done 2024-03-29T11:03:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a4c001c50 con 0x7f7a48030660 2024-03-29T11:03:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a45ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7a4c003a40 con 0x7f7a48030660 2024-03-29T11:03:28.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.118+0000 7f7a45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.119+0000 7f7a45ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.121+0000 7f7a45ffb640 10 monclient: dump: 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:28.104 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.121+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f7a4c003ba0 con 0x7f7a48030660 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.119+0000 7f7a477fe640 4 set_mon_vals no callback set 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.122+0000 7f7a5622b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.123+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f7a4c084b70 con 0x7f7a48030660 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.123+0000 7f7a45ffb640 10 client.4411.objecter ms_dispatch 0x7f7a50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.123+0000 7f7a45ffb640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.123+0000 7f7a45ffb640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.123+0000 7f7a45ffb640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.123+0000 7f7a5622b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.123+0000 7f7a5622b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.123+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a30005080 con 0x7f7a48030660 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.123+0000 7f7a54a28640 10 monclient: get_auth_request con 0x7f7a3406deb0 auth_method 0 2024-03-29T11:03:28.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.124+0000 7f7a45ffb640 10 client.4411.objecter ms_handle_connect 0x7f7a3406deb0 2024-03-29T11:03:28.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.130+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a4c050280 con 0x7f7a48030660 2024-03-29T11:03:28.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.131+0000 7f7a45ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:28.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.131+0000 7f7a45ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:28.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.490+0000 7f7a5622b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:28.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.490+0000 7f7a5622b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:28.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.490+0000 7f7a5622b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:28.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.490+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a30005330 con 0x7f7a48030660 2024-03-29T11:03:28.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.492+0000 7f7a45ffb640 1 -- 172.21.15.107:0/3086099018 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f7a4c052380 con 0x7f7a48030660 2024-03-29T11:03:28.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.492+0000 7f7a45ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:28.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.492+0000 7f7a45ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:28.471 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:03:28.472 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:28.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.496+0000 7f7a5622b640 10 librados: watch_flush enter 2024-03-29T11:03:28.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.496+0000 7f7a5622b640 10 librados: watch_flush exit 2024-03-29T11:03:28.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.496+0000 7f7a5622b640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:03:28.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.496+0000 7f7a5622b640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:03:28.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.497+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 >> v1:172.21.15.107:6816/882194755 conn(0x7f7a3406deb0 legacy=0x7f7a34070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:28.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.497+0000 7f7a5622b640 10 monclient: shutdown 2024-03-29T11:03:28.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.497+0000 7f7a5622b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:28.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.497+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 >> v1:172.21.15.129:6789/0 conn(0x7f7a48030660 legacy=0x7f7a4802d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:28.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.498+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 shutdown_connections 2024-03-29T11:03:28.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.498+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 >> 172.21.15.107:0/3086099018 conn(0x7f7a48005c00 msgr2=0x7f7a48005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:28.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.498+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 shutdown_connections 2024-03-29T11:03:28.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.498+0000 7f7a5622b640 1 -- 172.21.15.107:0/3086099018 wait complete. 2024-03-29T11:03:28.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:28.502+0000 7f7a5622b640 1 librados: shutdown 2024-03-29T11:03:28.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:35.308 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.480+0000 7faf0287b640 10 monclient: get_monmap_and_config 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.480+0000 7faf0287b640 10 monclient: build_initial_monmap 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.480+0000 7faf0287b640 10 monclient: monmap: 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:03:35.481536+0000 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:03:35.481536+0000 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:35.460 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:35.461 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:03:35.461 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:03:35.461 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:35.462 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.482+0000 7faf0287b640 1 Processor -- start 2024-03-29T11:03:35.463 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.482+0000 7faf0287b640 1 -- start start 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.482+0000 7faf0287b640 10 monclient: init 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 10 monclient(hunting): picked mon.noname-a con 0x7faefc1457f0 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 10 monclient(hunting): picked mon.noname-b con 0x7faefc13f650 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 10 monclient(hunting): picked mon.noname-c con 0x7faefc0a9d70 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faefc0aaef0 con 0x7faefc1457f0 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faefc145bd0 con 0x7faefc0a9d70 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faefc145eb0 con 0x7faefc13f650 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.483+0000 7faf0287b640 10 monclient(hunting): authenticate will time out at 1146.751834s 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.484+0000 7faefb7fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7faefc13f650 0x7faefc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:58914/0 (socket says 172.21.15.107:58914) 2024-03-29T11:03:35.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.484+0000 7faefb7fe640 1 -- 172.21.15.107:0/4143535372 learned_addr learned my addr 172.21.15.107:0/4143535372 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:35.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.485+0000 7faefaffd640 1 -- 172.21.15.107:0/4143535372 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998393583 0 0) 0x7faefc145eb0 con 0x7faefc13f650 2024-03-29T11:03:35.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.485+0000 7faefaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:35.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.485+0000 7faefaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:35.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.485+0000 7faefaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:35.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.485+0000 7faefaffd640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:03:35.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.485+0000 7faefaffd640 1 -- 172.21.15.107:0/4143535372 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee8003520 con 0x7faefc13f650 2024-03-29T11:03:35.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 1 -- 172.21.15.107:0/4143535372 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 947246431 0 0) 0x7faee8003520 con 0x7faefc13f650 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 1 -- 172.21.15.107:0/4143535372 >> v1:172.21.15.129:6789/0 conn(0x7faefc0a9d70 legacy=0x7faefc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 1 -- 172.21.15.107:0/4143535372 >> v1:172.21.15.107:6789/0 conn(0x7faefc1457f0 legacy=0x7faefc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 1 monclient: found mon.noname-b 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 1 -- 172.21.15.107:0/4143535372 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc146340 con 0x7faefc13f650 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 10 monclient: _finish_auth 0 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 1 -- 172.21.15.107:0/4143535372 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7faeec001a10 con 0x7faefc13f650 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:35.468 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 10 monclient: dump: 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faf0287b640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:03:35.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faf0287b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:35.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.488+0000 7faefaffd640 1 -- 172.21.15.107:0/4143535372 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faeec001be0 con 0x7faefc13f650 2024-03-29T11:03:35.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.489+0000 7faefaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:35.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.489+0000 7faefaffd640 1 -- 172.21.15.107:0/4143535372 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7faeec001ff0 con 0x7faefc13f650 2024-03-29T11:03:35.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.489+0000 7faefaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:35.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.489+0000 7faefaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:35.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.489+0000 7faefaffd640 10 monclient: dump: 2024-03-29T11:03:35.470 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:35.470 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:35.470 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.489+0000 7faf0287b640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.489+0000 7faf0287b640 4 set_mon_vals no callback set 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.490+0000 7faf0287b640 10 monclient: shutdown 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.490+0000 7faf0287b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.490+0000 7faf0287b640 1 -- 172.21.15.107:0/4143535372 >> v1:172.21.15.107:6790/0 conn(0x7faefc13f650 legacy=0x7faefc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.490+0000 7faf0287b640 1 -- 172.21.15.107:0/4143535372 shutdown_connections 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.490+0000 7faf0287b640 1 -- 172.21.15.107:0/4143535372 >> 172.21.15.107:0/4143535372 conn(0x7faefc066a00 msgr2=0x7faefc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:35.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.490+0000 7faf0287b640 1 -- 172.21.15.107:0/4143535372 shutdown_connections 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.490+0000 7faf0287b640 1 -- 172.21.15.107:0/4143535372 wait complete. 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.491+0000 7faf0287b640 10 monclient: build_initial_monmap 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.491+0000 7faf0287b640 10 monclient: monmap: 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:35.473 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:03:35.474 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:03:35.474 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:35.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.492+0000 7faf0287b640 1 librados: starting msgr at 2024-03-29T11:03:35.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.492+0000 7faf0287b640 1 librados: starting objecter 2024-03-29T11:03:35.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.492+0000 7faf0287b640 1 Processor -- start 2024-03-29T11:03:35.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.492+0000 7faf0287b640 1 -- start start 2024-03-29T11:03:35.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.492+0000 7faf0287b640 1 librados: setting wanted keys 2024-03-29T11:03:35.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.492+0000 7faf0287b640 1 librados: calling monclient init 2024-03-29T11:03:35.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.492+0000 7faf0287b640 10 monclient: init 2024-03-29T11:03:35.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:35.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:35.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 10 monclient(hunting): picked mon.noname-c con 0x7faefc0a9d70 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:35.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 10 monclient(hunting): picked mon.noname-b con 0x7faefc13f650 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:35.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 10 monclient(hunting): picked mon.noname-a con 0x7faefc1457f0 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:35.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faefc1447b0 con 0x7faefc1457f0 2024-03-29T11:03:35.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faefc06dfc0 con 0x7faefc13f650 2024-03-29T11:03:35.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faefc144a00 con 0x7faefc0a9d70 2024-03-29T11:03:35.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:35.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.493+0000 7faf0287b640 10 monclient(hunting): authenticate will time out at 1146.761870s 2024-03-29T11:03:35.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faefbfff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7faefc0a9d70 0x7faefc140a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60634/0 (socket says 172.21.15.107:60634) 2024-03-29T11:03:35.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faefbfff640 1 -- 172.21.15.107:0/858145847 learned_addr learned my addr 172.21.15.107:0/858145847 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:35.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faef8ff9640 10 client.?.objecter ms_handle_connect 0x7faefc0a9d70 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faef8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663922120 0 0) 0x7faefc144a00 con 0x7faefc0a9d70 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faef8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faef8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faef8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faef8ff9640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.500+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faedc004eb0 con 0x7faefc0a9d70 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 client.?.objecter ms_handle_connect 0x7faefc13f650 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3829835216 0 0) 0x7faedc004eb0 con 0x7faefc0a9d70 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faefc144a00 con 0x7faefc0a9d70 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7faee4001980 con 0x7faefc0a9d70 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:35.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient(hunting): dump: 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:35.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 client.?.objecter ms_handle_connect 0x7faefc1457f0 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727544418 0 0) 0x7faefc06dfc0 con 0x7faefc13f650 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faedc004eb0 con 0x7faefc13f650 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4004582522 0 0) 0x7faefc144a00 con 0x7faefc0a9d70 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 >> v1:172.21.15.129:6789/0 conn(0x7faefc13f650 legacy=0x7faefc1415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 >> v1:172.21.15.107:6789/0 conn(0x7faefc1457f0 legacy=0x7faefc1eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 monclient: found mon.c 2024-03-29T11:03:35.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc06e3e0 con 0x7faefc0a9d70 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faef8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faf0287b640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faf0287b640 10 monclient: _renew_subs 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faf0287b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faefc071ed0 con 0x7faefc0a9d70 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faf0287b640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faf0287b640 10 monclient: _renew_subs 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faf0287b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faefc141360 con 0x7faefc0a9d70 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.501+0000 7faf0287b640 1 librados: init done 2024-03-29T11:03:35.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.503+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faee4001c40 con 0x7faefc0a9d70 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.503+0000 7faef8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.503+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7faee4003a40 con 0x7faefc0a9d70 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.503+0000 7faef8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.503+0000 7faef8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.503+0000 7faef8ff9640 10 monclient: dump: 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:35.489 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:35.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.503+0000 7faefa7fc640 4 set_mon_vals no callback set 2024-03-29T11:03:35.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7faee4003ba0 con 0x7faefc0a9d70 2024-03-29T11:03:35.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faf0287b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:35.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faf0287b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:35.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faf0287b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:35.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee0005080 con 0x7faefc0a9d70 2024-03-29T11:03:35.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7faee4084a20 con 0x7faefc0a9d70 2024-03-29T11:03:35.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faef8ff9640 10 client.4487.objecter ms_dispatch 0x7faefc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:35.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faef8ff9640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:35.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faef8ff9640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:35.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faef8ff9640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T11:03:35.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.506+0000 7faefb7fe640 10 monclient: get_auth_request con 0x7faedc06dce0 auth_method 0 2024-03-29T11:03:35.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.507+0000 7faef8ff9640 10 client.4487.objecter ms_handle_connect 0x7faedc06dce0 2024-03-29T11:03:35.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.514+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faee4050300 con 0x7faefc0a9d70 2024-03-29T11:03:35.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.514+0000 7faef8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:35.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.514+0000 7faef8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.838+0000 7faf0287b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:35.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.838+0000 7faf0287b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.838+0000 7faf0287b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:35.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.838+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faee0005330 con 0x7faefc0a9d70 2024-03-29T11:03:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.841+0000 7faef8ff9640 1 -- 172.21.15.107:0/858145847 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7faee4052230 con 0x7faefc0a9d70 2024-03-29T11:03:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.841+0000 7faef8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:35.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.841+0000 7faef8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:35.821 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:03:35.821 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:35.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.845+0000 7faf0287b640 10 librados: watch_flush enter 2024-03-29T11:03:35.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.847+0000 7faf0287b640 10 librados: watch_flush exit 2024-03-29T11:03:35.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.847+0000 7faf0287b640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T11:03:35.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.847+0000 7faf0287b640 10 client.4487.objecter successfully canceled tick 2024-03-29T11:03:35.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.847+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 >> v1:172.21.15.107:6816/882194755 conn(0x7faedc06dce0 legacy=0x7faedc0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:35.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.847+0000 7faf0287b640 10 monclient: shutdown 2024-03-29T11:03:35.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.847+0000 7faf0287b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:35.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.847+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 >> v1:172.21.15.107:6790/0 conn(0x7faefc0a9d70 legacy=0x7faefc140a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:35.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.847+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 shutdown_connections 2024-03-29T11:03:35.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.847+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 >> 172.21.15.107:0/858145847 conn(0x7faefc066a00 msgr2=0x7faefc1da870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:35.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.848+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 shutdown_connections 2024-03-29T11:03:35.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.851+0000 7faf0287b640 1 -- 172.21.15.107:0/858145847 wait complete. 2024-03-29T11:03:35.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:35.852+0000 7faf0287b640 1 librados: shutdown 2024-03-29T11:03:35.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:42.657 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:42.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.817+0000 7f43ebbf4640 10 monclient: get_monmap_and_config 2024-03-29T11:03:42.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.817+0000 7f43ebbf4640 10 monclient: build_initial_monmap 2024-03-29T11:03:42.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.817+0000 7f43ebbf4640 10 monclient: monmap: 2024-03-29T11:03:42.797 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:42.797 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:42.798 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:03:42.818726+0000 2024-03-29T11:03:42.798 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:03:42.818726+0000 2024-03-29T11:03:42.798 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:42.798 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:42.798 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:42.798 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:03:42.798 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:03:42.798 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.818+0000 7f43ebbf4640 1 Processor -- start 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.818+0000 7f43ebbf4640 1 -- start start 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.818+0000 7f43ebbf4640 10 monclient: init 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 10 monclient(hunting): picked mon.noname-b con 0x7f43e40b1f60 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 10 monclient(hunting): picked mon.noname-c con 0x7f43e4071eb0 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 10 monclient(hunting): picked mon.noname-a con 0x7f43e4143d10 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e406ef80 con 0x7f43e4143d10 2024-03-29T11:03:42.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e406f260 con 0x7f43e4071eb0 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e40b40c0 con 0x7f43e40b1f60 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43ebbf4640 10 monclient(hunting): authenticate will time out at 1154.087762s 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43e9990640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f43e40b1f60 0x7f43e4143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34860/0 (socket says 172.21.15.107:34860) 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.819+0000 7f43e9990640 1 -- 172.21.15.107:0/3349920896 learned_addr learned my addr 172.21.15.107:0/3349920896 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328535643 0 0) 0x7f43e406ef80 con 0x7f43e4143d10 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 10 monclient(hunting): _negotiate 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d0003520 con 0x7f43e4143d10 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702813908 0 0) 0x7f43e40b40c0 con 0x7f43e40b1f60 2024-03-29T11:03:42.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 10 monclient(hunting): _negotiate 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e406ef80 con 0x7f43e40b1f60 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2273618810 0 0) 0x7f43d0003520 con 0x7f43e4143d10 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 >> v1:172.21.15.107:6790/0 conn(0x7f43e40b1f60 legacy=0x7f43e4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.820+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 >> v1:172.21.15.129:6789/0 conn(0x7f43e4071eb0 legacy=0x7f43e4146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 10 monclient: _finish_hunting 0 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 1 monclient: found mon.noname-a 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e4155d70 con 0x7f43e4143d10 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 10 monclient: _finish_auth 0 2024-03-29T11:03:42.805 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 10 monclient: _check_auth_tickets 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43ebbf4640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43ebbf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f43dc001a10 con 0x7f43e4143d10 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.821+0000 7f43e898e640 10 monclient: dump: 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:42.806 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.822+0000 7f43ebbf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.822+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43dc001c60 con 0x7f43e4143d10 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.822+0000 7f43e898e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:42.807 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.822+0000 7f43ebbf4640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:42.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.822+0000 7f43ebbf4640 4 set_mon_vals no callback set 2024-03-29T11:03:42.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.822+0000 7f43e898e640 1 -- 172.21.15.107:0/3349920896 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f43dc001ff0 con 0x7f43e4143d10 2024-03-29T11:03:42.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.822+0000 7f43e898e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:42.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.822+0000 7f43e898e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:42.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.822+0000 7f43e898e640 10 monclient: dump: 2024-03-29T11:03:42.808 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:42.808 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:42.809 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.809 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.809 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:42.809 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:42.809 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:42.809 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:42.810 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:42.810 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:42.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.823+0000 7f43ebbf4640 10 monclient: shutdown 2024-03-29T11:03:42.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.823+0000 7f43ebbf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:42.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.823+0000 7f43ebbf4640 1 -- 172.21.15.107:0/3349920896 >> v1:172.21.15.107:6789/0 conn(0x7f43e4143d10 legacy=0x7f43e4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:42.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.823+0000 7f43ebbf4640 1 -- 172.21.15.107:0/3349920896 shutdown_connections 2024-03-29T11:03:42.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.823+0000 7f43ebbf4640 1 -- 172.21.15.107:0/3349920896 >> 172.21.15.107:0/3349920896 conn(0x7f43e4066a00 msgr2=0x7f43e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:42.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.824+0000 7f43ebbf4640 1 -- 172.21.15.107:0/3349920896 shutdown_connections 2024-03-29T11:03:42.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.824+0000 7f43ebbf4640 1 -- 172.21.15.107:0/3349920896 wait complete. 2024-03-29T11:03:42.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.824+0000 7f43ebbf4640 10 monclient: build_initial_monmap 2024-03-29T11:03:42.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.824+0000 7f43ebbf4640 10 monclient: monmap: 2024-03-29T11:03:42.816 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:42.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 1 librados: starting msgr at 2024-03-29T11:03:42.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 1 librados: starting objecter 2024-03-29T11:03:42.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 1 Processor -- start 2024-03-29T11:03:42.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 1 -- start start 2024-03-29T11:03:42.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 1 librados: setting wanted keys 2024-03-29T11:03:42.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 1 librados: calling monclient init 2024-03-29T11:03:42.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 10 monclient: init 2024-03-29T11:03:42.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 10 monclient(hunting): picked mon.noname-b con 0x7f43e40b1f60 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 10 monclient(hunting): picked mon.noname-c con 0x7f43e4143d10 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43e918f640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f43e4143d10 0x7f43e412e8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34930/0 (socket says 172.21.15.107:34930) 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43e918f640 1 -- 172.21.15.107:0/894045688 learned_addr learned my addr 172.21.15.107:0/894045688 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 10 monclient(hunting): picked mon.noname-a con 0x7f43e4141d20 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e4070af0 con 0x7f43e4141d20 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e412f710 con 0x7f43e40b1f60 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e40707c0 con 0x7f43e4143d10 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.825+0000 7f43ebbf4640 10 monclient(hunting): authenticate will time out at 1154.093949s 2024-03-29T11:03:42.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 client.?.objecter ms_handle_connect 0x7f43e4141d20 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 client.?.objecter ms_handle_connect 0x7f43e4143d10 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558561997 0 0) 0x7f43e40707c0 con 0x7f43e4143d10 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c8004eb0 con 0x7f43e4143d10 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224338766 0 0) 0x7f43e4070af0 con 0x7f43e4141d20 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:03:42.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.826+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e40707c0 con 0x7f43e4141d20 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2616651584 0 0) 0x7f43c8004eb0 con 0x7f43e4143d10 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e4070af0 con 0x7f43e4143d10 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3892353676 0 0) 0x7f43e40707c0 con 0x7f43e4141d20 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43c8004eb0 con 0x7f43e4141d20 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f43e00017a0 con 0x7f43e4143d10 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 10 monclient(hunting): dump: 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:42.821 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f43dc001dd0 con 0x7f43e4141d20 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 10 monclient(hunting): dump: 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.822 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4190798181 0 0) 0x7f43c8004eb0 con 0x7f43e4141d20 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 >> v1:172.21.15.107:6790/0 conn(0x7f43e4143d10 legacy=0x7f43e412e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.827+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 >> v1:172.21.15.129:6789/0 conn(0x7f43e40b1f60 legacy=0x7f43e4155090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 1 monclient: found mon.a 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e4154260 con 0x7f43e4141d20 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:42.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43ebbf4640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43ebbf4640 10 monclient: _renew_subs 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43ebbf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e4154540 con 0x7f43e4141d20 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43dc0020b0 con 0x7f43e4141d20 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f43dc003dc0 con 0x7f43e4141d20 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43da7fc640 10 monclient: dump: 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.824 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43dbfff640 4 set_mon_vals no callback set 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43ebbf4640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43ebbf4640 10 monclient: _renew_subs 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43ebbf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.829+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e4154b00 con 0x7f43e4141d20 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.830+0000 7f43ebbf4640 1 librados: init done 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.831+0000 7f43ebbf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.831+0000 7f43ebbf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:42.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.831+0000 7f43ebbf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:42.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.831+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e4075f10 con 0x7f43e4141d20 2024-03-29T11:03:42.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.831+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f43dc019d90 con 0x7f43e4141d20 2024-03-29T11:03:42.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.832+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f43dc083a30 con 0x7f43e4141d20 2024-03-29T11:03:42.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.832+0000 7f43da7fc640 10 client.4443.objecter ms_dispatch 0x7f43e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:42.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.832+0000 7f43da7fc640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:42.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.832+0000 7f43da7fc640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:42.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.832+0000 7f43da7fc640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-29T11:03:42.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.832+0000 7f43e9990640 10 monclient: get_auth_request con 0x7f43c806dff0 auth_method 0 2024-03-29T11:03:42.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.833+0000 7f43da7fc640 10 client.4443.objecter ms_handle_connect 0x7f43c806dff0 2024-03-29T11:03:42.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.840+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43dc04f310 con 0x7f43e4141d20 2024-03-29T11:03:42.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.840+0000 7f43da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:42.827 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:42.840+0000 7f43da7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:43.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.150+0000 7f43ebbf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:43.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.150+0000 7f43ebbf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:43.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.150+0000 7f43ebbf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:03:43.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.150+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43e4074f50 con 0x7f43e4141d20 2024-03-29T11:03:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.151+0000 7f43da7fc640 1 -- 172.21.15.107:0/894045688 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f43dc051240 con 0x7f43e4141d20 2024-03-29T11:03:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.151+0000 7f43da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:43.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.151+0000 7f43da7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:43.131 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:03:43.131 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.154+0000 7f43ebbf4640 10 librados: watch_flush enter 2024-03-29T11:03:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.154+0000 7f43ebbf4640 10 librados: watch_flush exit 2024-03-29T11:03:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.154+0000 7f43ebbf4640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-29T11:03:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.154+0000 7f43ebbf4640 10 client.4443.objecter successfully canceled tick 2024-03-29T11:03:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.154+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 >> v1:172.21.15.107:6816/882194755 conn(0x7f43c806dff0 legacy=0x7f43c80704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.154+0000 7f43ebbf4640 10 monclient: shutdown 2024-03-29T11:03:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.155+0000 7f43ebbf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.155+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 >> v1:172.21.15.107:6789/0 conn(0x7f43e4141d20 legacy=0x7f43e412f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:43.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.155+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 shutdown_connections 2024-03-29T11:03:43.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.155+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 >> 172.21.15.107:0/894045688 conn(0x7f43e4066a00 msgr2=0x7f43e4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:43.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.155+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 shutdown_connections 2024-03-29T11:03:43.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.155+0000 7f43ebbf4640 1 -- 172.21.15.107:0/894045688 wait complete. 2024-03-29T11:03:43.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:43.155+0000 7f43ebbf4640 1 librados: shutdown 2024-03-29T11:03:43.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:49.955 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.123+0000 7f2bf2bac640 10 monclient: get_monmap_and_config 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.123+0000 7f2bf2bac640 10 monclient: build_initial_monmap 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.123+0000 7f2bf2bac640 10 monclient: monmap: 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:03:50.124475+0000 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:03:50.124475+0000 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:50.103 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:50.104 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:03:50.104 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:03:50.104 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:50.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.124+0000 7f2bf2bac640 1 Processor -- start 2024-03-29T11:03:50.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.124+0000 7f2bf2bac640 1 -- start start 2024-03-29T11:03:50.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.124+0000 7f2bf2bac640 10 monclient: init 2024-03-29T11:03:50.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.125+0000 7f2bf2bac640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:50.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.125+0000 7f2bf2bac640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:50.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.125+0000 7f2bf2bac640 10 monclient(hunting): picked mon.noname-b con 0x7f2bec1457f0 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:50.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2bf2bac640 10 monclient(hunting): picked mon.noname-c con 0x7f2bec13f650 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:50.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2bf0948640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f2bec1457f0 0x7f2bec13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40648/0 (socket says 172.21.15.107:40648) 2024-03-29T11:03:50.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2bf0948640 1 -- 172.21.15.107:0/803393493 learned_addr learned my addr 172.21.15.107:0/803393493 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2bf2bac640 10 monclient(hunting): picked mon.noname-a con 0x7f2bec0a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2bf2bac640 1 -- 172.21.15.107:0/803393493 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec0aaef0 con 0x7f2bec0a9d70 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2bf2bac640 1 -- 172.21.15.107:0/803393493 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec145bd0 con 0x7f2bec13f650 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2bf2bac640 1 -- 172.21.15.107:0/803393493 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec145eb0 con 0x7f2bec1457f0 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2bf2bac640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2bf2bac640 10 monclient(hunting): authenticate will time out at 1161.394504s 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239887746 0 0) 0x7f2bec145eb0 con 0x7f2bec1457f0 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2beb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2beb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2beb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2beb7fe640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.126+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bdc003520 con 0x7f2bec1457f0 2024-03-29T11:03:50.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193078867 0 0) 0x7f2bec0aaef0 con 0x7f2bec0a9d70 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bec145eb0 con 0x7f2bec0a9d70 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3306128349 0 0) 0x7f2bdc003520 con 0x7f2bec1457f0 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 >> v1:172.21.15.129:6789/0 conn(0x7f2bec13f650 legacy=0x7f2bec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 >> v1:172.21.15.107:6789/0 conn(0x7f2bec0a9d70 legacy=0x7f2bec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:50.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec146340 con 0x7f2bec1457f0 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2be0001690 con 0x7f2bec1457f0 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2beb7fe640 10 monclient: dump: 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:50.111 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2bf2bac640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.127+0000 7f2bf2bac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.128+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2be00017f0 con 0x7f2bec1457f0 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.128+0000 7f2beb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.128+0000 7f2beb7fe640 1 -- 172.21.15.107:0/803393493 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2be0001c40 con 0x7f2bec1457f0 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.128+0000 7f2beb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.128+0000 7f2beb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.128+0000 7f2beb7fe640 10 monclient: dump: 2024-03-29T11:03:50.112 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.128+0000 7f2bf2bac640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:50.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.128+0000 7f2bf2bac640 4 set_mon_vals no callback set 2024-03-29T11:03:50.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.129+0000 7f2bf2bac640 10 monclient: shutdown 2024-03-29T11:03:50.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.129+0000 7f2bf2bac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:50.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.129+0000 7f2bf2bac640 1 -- 172.21.15.107:0/803393493 >> v1:172.21.15.107:6790/0 conn(0x7f2bec1457f0 legacy=0x7f2bec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:50.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.130+0000 7f2bf2bac640 1 -- 172.21.15.107:0/803393493 shutdown_connections 2024-03-29T11:03:50.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.130+0000 7f2bf2bac640 1 -- 172.21.15.107:0/803393493 >> 172.21.15.107:0/803393493 conn(0x7f2bec066a00 msgr2=0x7f2bec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:50.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.130+0000 7f2bf2bac640 1 -- 172.21.15.107:0/803393493 shutdown_connections 2024-03-29T11:03:50.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.130+0000 7f2bf2bac640 1 -- 172.21.15.107:0/803393493 wait complete. 2024-03-29T11:03:50.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.131+0000 7f2bf2bac640 10 monclient: build_initial_monmap 2024-03-29T11:03:50.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.131+0000 7f2bf2bac640 10 monclient: monmap: 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.131+0000 7f2bf2bac640 1 librados: starting msgr at 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.131+0000 7f2bf2bac640 1 librados: starting objecter 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.132+0000 7f2bf2bac640 1 Processor -- start 2024-03-29T11:03:50.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.132+0000 7f2bf2bac640 1 -- start start 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.132+0000 7f2bf2bac640 1 librados: setting wanted keys 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.132+0000 7f2bf2bac640 1 librados: calling monclient init 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.132+0000 7f2bf2bac640 10 monclient: init 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 10 monclient(hunting): picked mon.noname-b con 0x7f2bec0a9d70 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 10 monclient(hunting): picked mon.noname-c con 0x7f2bec13f650 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 10 monclient(hunting): picked mon.noname-a con 0x7f2bec1457f0 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec1cb860 con 0x7f2bec1457f0 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bebfff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f2bec13f650 0x7f2bec138a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40932/0 (socket says 172.21.15.107:40932) 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bebfff640 1 -- 172.21.15.107:0/1179362946 learned_addr learned my addr 172.21.15.107:0/1179362946 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec1ccc80 con 0x7f2bec0a9d70 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec1ccf60 con 0x7f2bec13f650 2024-03-29T11:03:50.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2bf2bac640 10 monclient(hunting): authenticate will time out at 1161.401571s 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2be97fa640 10 client.?.objecter ms_handle_connect 0x7f2bec13f650 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.133+0000 7f2be97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.134+0000 7f2be97fa640 10 client.?.objecter ms_handle_connect 0x7f2bec1457f0 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.134+0000 7f2be97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.134+0000 7f2be97fa640 10 client.?.objecter ms_handle_connect 0x7f2bec0a9d70 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.134+0000 7f2be97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.134+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415513435 0 0) 0x7f2bec1ccf60 con 0x7f2bec13f650 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.134+0000 7f2be97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.134+0000 7f2be97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.134+0000 7f2be97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.134+0000 7f2be97fa640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:03:50.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd0004eb0 con 0x7f2bec13f650 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657315340 0 0) 0x7f2bec1cb860 con 0x7f2bec1457f0 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bec1ccf60 con 0x7f2bec1457f0 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231792967 0 0) 0x7f2bec1ccc80 con 0x7f2bec0a9d70 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bec1cb860 con 0x7f2bec0a9d70 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657808273 0 0) 0x7f2bd0004eb0 con 0x7f2bec13f650 2024-03-29T11:03:50.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bec1ccc80 con 0x7f2bec13f650 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1689047276 0 0) 0x7f2bec1ccf60 con 0x7f2bec1457f0 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.135+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bd0004eb0 con 0x7f2bec1457f0 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2932777410 0 0) 0x7f2bec1cb860 con 0x7f2bec0a9d70 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bec1ccf60 con 0x7f2bec0a9d70 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2bd8001990 con 0x7f2bec13f650 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): dump: 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:50.130 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2be4001c30 con 0x7f2bec1457f0 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): dump: 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.131 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2be0001b00 con 0x7f2bec0a9d70 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient(hunting): dump: 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.132 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669578616 0 0) 0x7f2bec1ccc80 con 0x7f2bec13f650 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 >> v1:172.21.15.129:6789/0 conn(0x7f2bec0a9d70 legacy=0x7f2bec13b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 >> v1:172.21.15.107:6789/0 conn(0x7f2bec1457f0 legacy=0x7f2bec1ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 monclient: found mon.c 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec1cb340 con 0x7f2bec13f650 2024-03-29T11:03:50.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.136+0000 7f2be97fa640 10 monclient: _finish_auth 0 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2be97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2be97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2bf2bac640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2bf2bac640 10 monclient: _renew_subs 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2bf2bac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bec1cb590 con 0x7f2bec13f650 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2bf2bac640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2bf2bac640 10 monclient: _renew_subs 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2bf2bac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bec1cbed0 con 0x7f2bec13f650 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2bf2bac640 1 librados: init done 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bd8001af0 con 0x7f2bec13f650 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2be97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:50.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.137+0000 7f2beaffd640 4 set_mon_vals no callback set 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.138+0000 7f2bf2bac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.138+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2bd8001d30 con 0x7f2bec13f650 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.139+0000 7f2bf2bac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.139+0000 7f2bf2bac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.139+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc005080 con 0x7f2bec13f650 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.139+0000 7f2be97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.139+0000 7f2be97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.139+0000 7f2be97fa640 10 monclient: dump: 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:50.135 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.140+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f2bd80038c0 con 0x7f2bec13f650 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.141+0000 7f2bf0948640 10 monclient: get_auth_request con 0x7f2bd006e150 auth_method 0 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.141+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f2bd80848c0 con 0x7f2bec13f650 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.141+0000 7f2be97fa640 10 client.4499.objecter ms_dispatch 0x7f2bec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.141+0000 7f2be97fa640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.141+0000 7f2be97fa640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.141+0000 7f2be97fa640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:03:50.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.141+0000 7f2be97fa640 10 client.4499.objecter ms_handle_connect 0x7f2bd006e150 2024-03-29T11:03:50.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.148+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bd80501a0 con 0x7f2bec13f650 2024-03-29T11:03:50.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.148+0000 7f2be97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:50.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.148+0000 7f2be97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:50.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.505+0000 7f2bf2bac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:50.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.505+0000 7f2bf2bac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:50.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.505+0000 7f2bf2bac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:03:50.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.505+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bcc005330 con 0x7f2bec13f650 2024-03-29T11:03:50.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.512+0000 7f2be97fa640 1 -- 172.21.15.107:0/1179362946 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f2bd80520d0 con 0x7f2bec13f650 2024-03-29T11:03:50.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.512+0000 7f2be97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:50.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.512+0000 7f2be97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:50.492 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:03:50.492 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:50.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.516+0000 7f2bf2bac640 10 librados: watch_flush enter 2024-03-29T11:03:50.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.516+0000 7f2bf2bac640 10 librados: watch_flush exit 2024-03-29T11:03:50.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.516+0000 7f2bf2bac640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:03:50.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.516+0000 7f2bf2bac640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:03:50.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.516+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 >> v1:172.21.15.107:6816/882194755 conn(0x7f2bd006e150 legacy=0x7f2bd0070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:50.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.516+0000 7f2bf2bac640 10 monclient: shutdown 2024-03-29T11:03:50.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.516+0000 7f2bf2bac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:50.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.516+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 >> v1:172.21.15.107:6790/0 conn(0x7f2bec13f650 legacy=0x7f2bec138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:50.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.517+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 shutdown_connections 2024-03-29T11:03:50.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.517+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 >> 172.21.15.107:0/1179362946 conn(0x7f2bec066a00 msgr2=0x7f2bec0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:50.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.517+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 shutdown_connections 2024-03-29T11:03:50.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.517+0000 7f2bf2bac640 1 -- 172.21.15.107:0/1179362946 wait complete. 2024-03-29T11:03:50.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:50.517+0000 7f2bf2bac640 1 librados: shutdown 2024-03-29T11:03:50.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:57.319 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:57.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.487+0000 7f2baf2e4640 10 monclient: get_monmap_and_config 2024-03-29T11:03:57.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.487+0000 7f2baf2e4640 10 monclient: build_initial_monmap 2024-03-29T11:03:57.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.487+0000 7f2baf2e4640 10 monclient: monmap: 2024-03-29T11:03:57.467 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:57.467 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:57.467 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:03:57.488375+0000 2024-03-29T11:03:57.468 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:03:57.488375+0000 2024-03-29T11:03:57.468 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:57.468 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:57.468 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:57.468 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:03:57.468 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:03:57.468 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:57.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.488+0000 7f2baf2e4640 1 Processor -- start 2024-03-29T11:03:57.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.488+0000 7f2baf2e4640 1 -- start start 2024-03-29T11:03:57.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.488+0000 7f2baf2e4640 10 monclient: init 2024-03-29T11:03:57.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:57.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:57.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f2ba80b0770 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:57.470 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba814a700 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:57.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f2ba80ac5a0 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:57.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba80aba70 con 0x7f2ba814a700 2024-03-29T11:03:57.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba80aecb0 con 0x7f2ba80b0770 2024-03-29T11:03:57.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba80af150 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:57.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2baf2e4640 10 monclient(hunting): authenticate will time out at 1168.757457s 2024-03-29T11:03:57.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2bad881640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f2ba80ac5a0 0x7f2ba8151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54674/0 (socket says 172.21.15.107:54674) 2024-03-29T11:03:57.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.489+0000 7f2bad881640 1 -- 172.21.15.107:0/1047235868 learned_addr learned my addr 172.21.15.107:0/1047235868 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:57.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.491+0000 7f2b9ffff640 1 -- 172.21.15.107:0/1047235868 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306527637 0 0) 0x7f2ba80af150 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.491+0000 7f2b9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:57.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.491+0000 7f2b9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:57.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.491+0000 7f2b9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:57.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.491+0000 7f2b9ffff640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:03:57.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.491+0000 7f2b9ffff640 1 -- 172.21.15.107:0/1047235868 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b94003520 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.492+0000 7f2b9ffff640 1 -- 172.21.15.107:0/1047235868 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2999785220 0 0) 0x7f2b94003520 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.492+0000 7f2b9ffff640 1 -- 172.21.15.107:0/1047235868 >> v1:172.21.15.129:6789/0 conn(0x7f2ba80b0770 legacy=0x7f2ba8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:03:57.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.492+0000 7f2b9ffff640 1 -- 172.21.15.107:0/1047235868 >> v1:172.21.15.107:6789/0 conn(0x7f2ba814a700 legacy=0x7f2ba80abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 1 monclient: found mon.noname-b 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 1 -- 172.21.15.107:0/1047235868 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba80ac9d0 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 10 monclient: _finish_auth 0 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 1 -- 172.21.15.107:0/1047235868 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2ba0001a10 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2b9ffff640 10 monclient: dump: 2024-03-29T11:03:57.476 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2baf2e4640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:03:57.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.495+0000 7f2baf2e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.498+0000 7f2b9ffff640 1 -- 172.21.15.107:0/1047235868 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ba0001be0 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.498+0000 7f2b9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.498+0000 7f2b9ffff640 1 -- 172.21.15.107:0/1047235868 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2ba0002030 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.498+0000 7f2b9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.498+0000 7f2b9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.498+0000 7f2b9ffff640 10 monclient: dump: 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.478 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:57.479 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:57.479 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:57.479 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:57.479 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:57.479 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:57.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.498+0000 7f2baf2e4640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:57.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.498+0000 7f2baf2e4640 4 set_mon_vals no callback set 2024-03-29T11:03:57.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.499+0000 7f2baf2e4640 10 monclient: shutdown 2024-03-29T11:03:57.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.499+0000 7f2baf2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:57.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.499+0000 7f2baf2e4640 1 -- 172.21.15.107:0/1047235868 >> v1:172.21.15.107:6790/0 conn(0x7f2ba80ac5a0 legacy=0x7f2ba8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:57.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.500+0000 7f2baf2e4640 1 -- 172.21.15.107:0/1047235868 shutdown_connections 2024-03-29T11:03:57.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.500+0000 7f2baf2e4640 1 -- 172.21.15.107:0/1047235868 >> 172.21.15.107:0/1047235868 conn(0x7f2ba806b660 msgr2=0x7f2ba806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:57.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.501+0000 7f2baf2e4640 1 -- 172.21.15.107:0/1047235868 shutdown_connections 2024-03-29T11:03:57.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.502+0000 7f2baf2e4640 1 -- 172.21.15.107:0/1047235868 wait complete. 2024-03-29T11:03:57.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.503+0000 7f2baf2e4640 10 monclient: build_initial_monmap 2024-03-29T11:03:57.482 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.503+0000 7f2baf2e4640 10 monclient: monmap: 2024-03-29T11:03:57.482 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:03:57.482 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:57.482 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:03:57.483 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:03:57.483 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:57.483 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:57.483 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:03:57.483 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:03:57.483 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:03:57.483 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.503+0000 7f2baf2e4640 1 librados: starting msgr at 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.503+0000 7f2baf2e4640 1 librados: starting objecter 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.503+0000 7f2baf2e4640 1 Processor -- start 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.503+0000 7f2baf2e4640 1 -- start start 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.503+0000 7f2baf2e4640 1 librados: setting wanted keys 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.503+0000 7f2baf2e4640 1 librados: calling monclient init 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.503+0000 7f2baf2e4640 10 monclient: init 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f2ba80ac5a0 addr v1:172.21.15.107:6790/0 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba80b0770 addr v1:172.21.15.107:6789/0 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f2ba814a700 addr v1:172.21.15.129:6789/0 2024-03-29T11:03:57.484 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba81285d0 con 0x7f2ba80b0770 2024-03-29T11:03:57.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba8129090 con 0x7f2ba814a700 2024-03-29T11:03:57.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba8129220 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:57.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.504+0000 7f2baf2e4640 10 monclient(hunting): authenticate will time out at 1168.772494s 2024-03-29T11:03:57.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.506+0000 7f2bad080640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f2ba80ac5a0 0x7f2ba8146ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:57548/0 (socket says 172.21.15.107:57548) 2024-03-29T11:03:57.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.506+0000 7f2bad080640 1 -- 172.21.15.107:0/4076529703 learned_addr learned my addr 172.21.15.107:0/4076529703 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:03:57.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 client.?.objecter ms_handle_connect 0x7f2ba80ac5a0 2024-03-29T11:03:57.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:57.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 client.?.objecter ms_handle_connect 0x7f2ba814a700 2024-03-29T11:03:57.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:57.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807112036 0 0) 0x7f2ba8129220 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:03:57.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:57.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b8c004eb0 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411891686 0 0) 0x7f2ba8129090 con 0x7f2ba814a700 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.507+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba8129220 con 0x7f2ba814a700 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3103538666 0 0) 0x7f2ba8129220 con 0x7f2ba814a700 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ba8129090 con 0x7f2ba814a700 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2ba0001ef0 con 0x7f2ba814a700 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 10 monclient(hunting): dump: 2024-03-29T11:03:57.488 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2694960281 0 0) 0x7f2b8c004eb0 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b8c0069a0 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2b98001960 con 0x7f2ba80ac5a0 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 10 monclient(hunting): dump: 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:57.490 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1152067782 0 0) 0x7f2ba8129090 con 0x7f2ba814a700 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 >> v1:172.21.15.107:6790/0 conn(0x7f2ba80ac5a0 legacy=0x7f2ba8146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.508+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 >> v1:172.21.15.107:6789/0 conn(0x7f2ba80b0770 legacy=0x7f2ba81deb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2b9dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2b9dffb640 1 monclient: found mon.b 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2b9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2b9dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba81475b0 con 0x7f2ba814a700 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2b9dffb640 10 monclient: _finish_auth 0 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2b9dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2b9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2baf2e4640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:03:57.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2baf2e4640 10 monclient: _renew_subs 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2baf2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.510+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba81477a0 con 0x7f2ba814a700 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.511+0000 7f2baf2e4640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.511+0000 7f2baf2e4640 10 monclient: _renew_subs 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.511+0000 7f2baf2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.511+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba8147c00 con 0x7f2ba814a700 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.511+0000 7f2baf2e4640 1 librados: init done 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.512+0000 7f2baf2e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.512+0000 7f2baf2e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.512+0000 7f2baf2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:57.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.512+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba80aafa0 con 0x7f2ba814a700 2024-03-29T11:03:57.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.515+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ba00020d0 con 0x7f2ba814a700 2024-03-29T11:03:57.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.515+0000 7f2b9dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.515+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f2ba0001ef0 con 0x7f2ba814a700 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.515+0000 7f2b9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.515+0000 7f2b9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.515+0000 7f2b9dffb640 10 monclient: dump: 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:03:57.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.515+0000 7f2b9effd640 4 set_mon_vals no callback set 2024-03-29T11:03:57.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.516+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f2ba001ade0 con 0x7f2ba814a700 2024-03-29T11:03:57.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.517+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f2ba0084ae0 con 0x7f2ba814a700 2024-03-29T11:03:57.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.517+0000 7f2b9dffb640 10 client.4423.objecter ms_dispatch 0x7f2ba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:57.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.517+0000 7f2b9dffb640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:57.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.517+0000 7f2b9dffb640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:57.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.517+0000 7f2b9dffb640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:03:57.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.517+0000 7f2bad080640 10 monclient: get_auth_request con 0x7f2b8c06e120 auth_method 0 2024-03-29T11:03:57.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.517+0000 7f2b9dffb640 10 client.4423.objecter ms_handle_connect 0x7f2b8c06e120 2024-03-29T11:03:57.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.522+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ba00501f0 con 0x7f2ba814a700 2024-03-29T11:03:57.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.522+0000 7f2b9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:57.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.522+0000 7f2b9dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:57.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.844+0000 7f2baf2e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:57.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.844+0000 7f2baf2e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:57.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.844+0000 7f2baf2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:03:57.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.844+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ba8075f10 con 0x7f2ba814a700 2024-03-29T11:03:57.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.845+0000 7f2b9dffb640 1 -- 172.21.15.107:0/4076529703 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f2ba00522f0 con 0x7f2ba814a700 2024-03-29T11:03:57.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.845+0000 7f2b9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:57.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.845+0000 7f2b9dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:57.825 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:03:57.825 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:57.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.848+0000 7f2baf2e4640 10 librados: watch_flush enter 2024-03-29T11:03:57.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.848+0000 7f2baf2e4640 10 librados: watch_flush exit 2024-03-29T11:03:57.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.848+0000 7f2baf2e4640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:03:57.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.849+0000 7f2baf2e4640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:03:57.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.849+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 >> v1:172.21.15.107:6816/882194755 conn(0x7f2b8c06e120 legacy=0x7f2b8c0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:57.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.849+0000 7f2baf2e4640 10 monclient: shutdown 2024-03-29T11:03:57.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.849+0000 7f2baf2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:57.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.849+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 >> v1:172.21.15.129:6789/0 conn(0x7f2ba814a700 legacy=0x7f2ba81e6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:57.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.851+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 shutdown_connections 2024-03-29T11:03:57.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.851+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 >> 172.21.15.107:0/4076529703 conn(0x7f2ba806b660 msgr2=0x7f2ba80b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:57.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.851+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 shutdown_connections 2024-03-29T11:03:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.852+0000 7f2baf2e4640 1 -- 172.21.15.107:0/4076529703 wait complete. 2024-03-29T11:03:57.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:03:57.852+0000 7f2baf2e4640 1 librados: shutdown 2024-03-29T11:03:57.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:04.655 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:04.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.826+0000 7fb4535a3640 10 monclient: get_monmap_and_config 2024-03-29T11:04:04.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.826+0000 7fb4535a3640 10 monclient: build_initial_monmap 2024-03-29T11:04:04.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.826+0000 7fb4535a3640 10 monclient: monmap: 2024-03-29T11:04:04.809 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:04.809 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:04.809 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:04:04.827636+0000 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:04:04.827636+0000 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.827+0000 7fb4535a3640 1 Processor -- start 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.828+0000 7fb4535a3640 1 -- start start 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.828+0000 7fb4535a3640 10 monclient: init 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:04:04.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 10 monclient(hunting): picked mon.noname-b con 0x7fb44c0ad6c0 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:04.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 10 monclient(hunting): picked mon.noname-a con 0x7fb44c0af6c0 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:04.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 10 monclient(hunting): picked mon.noname-c con 0x7fb44c153090 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:04.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0a94e0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0af3a0 con 0x7fb44c153090 2024-03-29T11:04:04.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0afb60 con 0x7fb44c0ad6c0 2024-03-29T11:04:04.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:04.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb4535a3640 10 monclient(hunting): authenticate will time out at 1176.097257s 2024-03-29T11:04:04.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb451b40640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb44c153090 0x7fb44c15a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:49356/0 (socket says 172.21.15.107:49356) 2024-03-29T11:04:04.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.829+0000 7fb451b40640 1 -- 172.21.15.107:0/2913858731 learned_addr learned my addr 172.21.15.107:0/2913858731 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:04.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438770421 0 0) 0x7fb44c0afb60 con 0x7fb44c0ad6c0 2024-03-29T11:04:04.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:04.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb438003520 con 0x7fb44c0ad6c0 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313677127 0 0) 0x7fb44c0af3a0 con 0x7fb44c153090 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4380048a0 con 0x7fb44c153090 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996752157 0 0) 0x7fb44c0a94e0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:04.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:04.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:04:04.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb438004be0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2690367797 0 0) 0x7fb4380048a0 con 0x7fb44c153090 2024-03-29T11:04:04.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 >> v1:172.21.15.107:6790/0 conn(0x7fb44c0ad6c0 legacy=0x7fb44c14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 >> v1:172.21.15.107:6789/0 conn(0x7fb44c0af6c0 legacy=0x7fb44c152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient: _finish_hunting 0 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 monclient: found mon.noname-c 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c0aab80 con 0x7fb44c153090 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient: _finish_auth 0 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient: _check_auth_tickets 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb43c0015f0 con 0x7fb44c153090 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.833+0000 7fb443fff640 10 monclient: dump: 2024-03-29T11:04:04.820 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.834+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb43c0017d0 con 0x7fb44c153090 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.834+0000 7fb443fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.834+0000 7fb443fff640 1 -- 172.21.15.107:0/2913858731 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb43c001c20 con 0x7fb44c153090 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.834+0000 7fb443fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:04.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.834+0000 7fb443fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.834+0000 7fb443fff640 10 monclient: dump: 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.834+0000 7fb4535a3640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:04:04.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.834+0000 7fb4535a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.834+0000 7fb4535a3640 4 set_mon_vals no callback set 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.835+0000 7fb4535a3640 10 monclient: shutdown 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.835+0000 7fb4535a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.835+0000 7fb4535a3640 1 -- 172.21.15.107:0/2913858731 >> v1:172.21.15.129:6789/0 conn(0x7fb44c153090 legacy=0x7fb44c15a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.835+0000 7fb4535a3640 1 -- 172.21.15.107:0/2913858731 shutdown_connections 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.835+0000 7fb4535a3640 1 -- 172.21.15.107:0/2913858731 >> 172.21.15.107:0/2913858731 conn(0x7fb44c0b35e0 msgr2=0x7fb44c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.836+0000 7fb4535a3640 1 -- 172.21.15.107:0/2913858731 shutdown_connections 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.836+0000 7fb4535a3640 1 -- 172.21.15.107:0/2913858731 wait complete. 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.838+0000 7fb4535a3640 10 monclient: build_initial_monmap 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.838+0000 7fb4535a3640 10 monclient: monmap: 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:04:04.823 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.838+0000 7fb4535a3640 1 librados: starting msgr at 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.838+0000 7fb4535a3640 1 librados: starting objecter 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.838+0000 7fb4535a3640 1 Processor -- start 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.838+0000 7fb4535a3640 1 -- start start 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.838+0000 7fb4535a3640 1 librados: setting wanted keys 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.838+0000 7fb4535a3640 1 librados: calling monclient init 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.838+0000 7fb4535a3640 10 monclient: init 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.839+0000 7fb4535a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.839+0000 7fb4535a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:04.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.839+0000 7fb4535a3640 10 monclient(hunting): picked mon.noname-b con 0x7fb44c0ad6c0 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:04.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.839+0000 7fb4535a3640 10 monclient(hunting): picked mon.noname-a con 0x7fb44c0af6c0 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:04.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.840+0000 7fb4535a3640 10 monclient(hunting): picked mon.noname-c con 0x7fb44c153090 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:04.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.840+0000 7fb4535a3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c071fe0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.840+0000 7fb4535a3640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c1e7620 con 0x7fb44c0ad6c0 2024-03-29T11:04:04.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.840+0000 7fb4535a3640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c13da70 con 0x7fb44c153090 2024-03-29T11:04:04.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.840+0000 7fb4535a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:04.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.840+0000 7fb4535a3640 10 monclient(hunting): authenticate will time out at 1176.108669s 2024-03-29T11:04:04.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.843+0000 7fb45133f640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fb44c0ad6c0 0x7fb44c0718d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:56038/0 (socket says 172.21.15.107:56038) 2024-03-29T11:04:04.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.843+0000 7fb45133f640 1 -- 172.21.15.107:0/995915982 learned_addr learned my addr 172.21.15.107:0/995915982 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:04.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 client.?.objecter ms_handle_connect 0x7fb44c0ad6c0 2024-03-29T11:04:04.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:04.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 client.?.objecter ms_handle_connect 0x7fb44c153090 2024-03-29T11:04:04.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:04.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013721806 0 0) 0x7fb44c1e7620 con 0x7fb44c0ad6c0 2024-03-29T11:04:04.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb430004eb0 con 0x7fb44c0ad6c0 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 client.?.objecter ms_handle_connect 0x7fb44c0af6c0 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268845983 0 0) 0x7fb44c071fe0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.844+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb44c1e7620 con 0x7fb44c0af6c0 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.845+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3436734609 0 0) 0x7fb430004eb0 con 0x7fb44c0ad6c0 2024-03-29T11:04:04.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.845+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb44c071fe0 con 0x7fb44c0ad6c0 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.845+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb444001d70 con 0x7fb44c0ad6c0 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.845+0000 7fb441ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.845+0000 7fb441ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.845+0000 7fb441ffb640 10 monclient(hunting): dump: 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:04.831 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.845+0000 7fb441ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1469154188 0 0) 0x7fb44c1e7620 con 0x7fb44c0af6c0 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb430004eb0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb4480018b0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient(hunting): dump: 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:04.832 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682439981 0 0) 0x7fb44c13da70 con 0x7fb44c153090 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb44c1e7620 con 0x7fb44c153090 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2647204580 0 0) 0x7fb430004eb0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 >> v1:172.21.15.107:6790/0 conn(0x7fb44c153090 legacy=0x7fb44c08a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 >> v1:172.21.15.129:6789/0 conn(0x7fb44c0ad6c0 legacy=0x7fb44c0718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 monclient: found mon.a 2024-03-29T11:04:04.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c06e7a0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient: _finish_auth 0 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.846+0000 7fb441ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb448003610 con 0x7fb44c0af6c0 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb441ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb448003920 con 0x7fb44c0af6c0 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb441ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb441ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb441ffb640 10 monclient: dump: 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:04.834 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb4535a3640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb4535a3640 10 monclient: _renew_subs 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb4535a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c1e8210 con 0x7fb44c0af6c0 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.847+0000 7fb4437fe640 4 set_mon_vals no callback set 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.848+0000 7fb4535a3640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:04.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.848+0000 7fb4535a3640 10 monclient: _renew_subs 2024-03-29T11:04:04.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.848+0000 7fb4535a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:04.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.848+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c070fc0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.848+0000 7fb4535a3640 1 librados: init done 2024-03-29T11:04:04.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.852+0000 7fb4535a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:04.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.852+0000 7fb4535a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:04.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.852+0000 7fb4535a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:04.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.852+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c13d890 con 0x7fb44c0af6c0 2024-03-29T11:04:04.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.854+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb4480198f0 con 0x7fb44c0af6c0 2024-03-29T11:04:04.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.855+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fb448083870 con 0x7fb44c0af6c0 2024-03-29T11:04:04.837 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.855+0000 7fb45133f640 10 monclient: get_auth_request con 0x7fb43006dfe0 auth_method 0 2024-03-29T11:04:04.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.855+0000 7fb441ffb640 10 client.4455.objecter ms_dispatch 0x7fb44c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:04.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.855+0000 7fb441ffb640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:04.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.855+0000 7fb441ffb640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:04.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.855+0000 7fb441ffb640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-29T11:04:04.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.855+0000 7fb441ffb640 10 client.4455.objecter ms_handle_connect 0x7fb43006dfe0 2024-03-29T11:04:04.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.863+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb44804ef80 con 0x7fb44c0af6c0 2024-03-29T11:04:04.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.863+0000 7fb441ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:04.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:04.863+0000 7fb441ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:05.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.183+0000 7fb4535a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:05.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.183+0000 7fb4535a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:05.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.183+0000 7fb4535a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:05.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.183+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb44c075f10 con 0x7fb44c0af6c0 2024-03-29T11:04:05.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.184+0000 7fb441ffb640 1 -- 172.21.15.107:0/995915982 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fb448051080 con 0x7fb44c0af6c0 2024-03-29T11:04:05.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.184+0000 7fb441ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:05.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.184+0000 7fb441ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:05.164 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:04:05.164 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:05.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.187+0000 7fb4535a3640 10 librados: watch_flush enter 2024-03-29T11:04:05.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.187+0000 7fb4535a3640 10 librados: watch_flush exit 2024-03-29T11:04:05.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.187+0000 7fb4535a3640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-29T11:04:05.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.187+0000 7fb4535a3640 10 client.4455.objecter successfully canceled tick 2024-03-29T11:04:05.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.188+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 >> v1:172.21.15.107:6816/882194755 conn(0x7fb43006dfe0 legacy=0x7fb4300704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:05.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.188+0000 7fb4535a3640 10 monclient: shutdown 2024-03-29T11:04:05.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.188+0000 7fb4535a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:05.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.188+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 >> v1:172.21.15.107:6789/0 conn(0x7fb44c0af6c0 legacy=0x7fb44c1e7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:05.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.191+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 shutdown_connections 2024-03-29T11:04:05.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.192+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 >> 172.21.15.107:0/995915982 conn(0x7fb44c0b35e0 msgr2=0x7fb44c14f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:05.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.192+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 shutdown_connections 2024-03-29T11:04:05.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.192+0000 7fb4535a3640 1 -- 172.21.15.107:0/995915982 wait complete. 2024-03-29T11:04:05.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:05.192+0000 7fb4535a3640 1 librados: shutdown 2024-03-29T11:04:05.189 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:11.997 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:12.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.167+0000 7f4d58dab640 10 monclient: get_monmap_and_config 2024-03-29T11:04:12.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.167+0000 7f4d58dab640 10 monclient: build_initial_monmap 2024-03-29T11:04:12.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.167+0000 7f4d58dab640 10 monclient: monmap: 2024-03-29T11:04:12.146 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:12.147 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:12.147 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:04:12.168000+0000 2024-03-29T11:04:12.147 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:04:12.168000+0000 2024-03-29T11:04:12.147 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:12.147 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:12.147 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:12.147 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:04:12.147 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:04:12.147 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:12.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.169+0000 7f4d58dab640 1 Processor -- start 2024-03-29T11:04:12.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.169+0000 7f4d58dab640 1 -- start start 2024-03-29T11:04:12.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.169+0000 7f4d58dab640 10 monclient: init 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4c038c60 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 10 monclient(hunting): picked mon.noname-a con 0x7f4d4c0324b0 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 10 monclient(hunting): picked mon.noname-b con 0x7f4d4c0d11e0 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c0caf80 con 0x7f4d4c0324b0 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c039850 con 0x7f4d4c038c60 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c0c78d0 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:12.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.170+0000 7f4d58dab640 10 monclient(hunting): authenticate will time out at 1183.438616s 2024-03-29T11:04:12.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.173+0000 7f4d53fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4d4c0d11e0 0x7f4d4c0d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41238/0 (socket says 172.21.15.107:41238) 2024-03-29T11:04:12.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.173+0000 7f4d53fff640 1 -- 172.21.15.107:0/84070665 learned_addr learned my addr 172.21.15.107:0/84070665 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:12.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 1 -- 172.21.15.107:0/84070665 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253489374 0 0) 0x7f4d4c0c78d0 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:12.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:12.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:12.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:04:12.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 1 -- 172.21.15.107:0/84070665 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d44003520 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 1 -- 172.21.15.107:0/84070665 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 611856017 0 0) 0x7f4d44003520 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 1 -- 172.21.15.107:0/84070665 >> v1:172.21.15.129:6789/0 conn(0x7f4d4c038c60 legacy=0x7f4d4c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:12.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 1 -- 172.21.15.107:0/84070665 >> v1:172.21.15.107:6789/0 conn(0x7f4d4c0324b0 legacy=0x7f4d4c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:04:12.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:04:12.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 1 monclient: found mon.noname-b 2024-03-29T11:04:12.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:12.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:04:12.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 1 -- 172.21.15.107:0/84070665 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c0c7a60 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 10 monclient: _finish_auth 0 2024-03-29T11:04:12.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.174+0000 7f4d527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 1 -- 172.21.15.107:0/84070665 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4d5404f730 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 10 monclient: dump: 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:12.159 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 1 -- 172.21.15.107:0/84070665 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d54075f10 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 1 -- 172.21.15.107:0/84070665 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4d540758b0 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.175+0000 7f4d527fc640 10 monclient: dump: 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:12.160 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.176+0000 7f4d58dab640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.176+0000 7f4d58dab640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.176+0000 7f4d58dab640 4 set_mon_vals no callback set 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.177+0000 7f4d58dab640 10 monclient: shutdown 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.178+0000 7f4d58dab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:12.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.178+0000 7f4d58dab640 1 -- 172.21.15.107:0/84070665 >> v1:172.21.15.107:6790/0 conn(0x7f4d4c0d11e0 legacy=0x7f4d4c0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:12.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.183+0000 7f4d58dab640 1 -- 172.21.15.107:0/84070665 shutdown_connections 2024-03-29T11:04:12.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.183+0000 7f4d58dab640 1 -- 172.21.15.107:0/84070665 >> 172.21.15.107:0/84070665 conn(0x7f4d4c005c20 msgr2=0x7f4d4c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:12.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.185+0000 7f4d58dab640 1 -- 172.21.15.107:0/84070665 shutdown_connections 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.186+0000 7f4d58dab640 1 -- 172.21.15.107:0/84070665 wait complete. 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.187+0000 7f4d58dab640 10 monclient: build_initial_monmap 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.187+0000 7f4d58dab640 10 monclient: monmap: 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:12.167 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.187+0000 7f4d58dab640 1 librados: starting msgr at 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.187+0000 7f4d58dab640 1 librados: starting objecter 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.187+0000 7f4d58dab640 1 Processor -- start 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.188+0000 7f4d58dab640 1 -- start start 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.188+0000 7f4d58dab640 1 librados: setting wanted keys 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.188+0000 7f4d58dab640 1 librados: calling monclient init 2024-03-29T11:04:12.168 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.188+0000 7f4d58dab640 10 monclient: init 2024-03-29T11:04:12.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.188+0000 7f4d58dab640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:12.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.189+0000 7f4d58dab640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:12.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.189+0000 7f4d58dab640 10 monclient(hunting): picked mon.noname-b con 0x7f4d4c0324b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:12.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.190+0000 7f4d58dab640 10 monclient(hunting): picked mon.noname-a con 0x7f4d4c038c60 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:12.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.191+0000 7f4d58dab640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4c0d11e0 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:12.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.191+0000 7f4d58dab640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c160840 con 0x7f4d4c038c60 2024-03-29T11:04:12.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.191+0000 7f4d58dab640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c160cd0 con 0x7f4d4c0324b0 2024-03-29T11:04:12.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.191+0000 7f4d58dab640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c036550 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.191+0000 7f4d58dab640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:12.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.191+0000 7f4d58dab640 10 monclient(hunting): authenticate will time out at 1183.459060s 2024-03-29T11:04:12.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.191+0000 7f4d53fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f4d4c0d11e0 0x7f4d4c0343d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:39176/0 (socket says 172.21.15.107:39176) 2024-03-29T11:04:12.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.191+0000 7f4d53fff640 1 -- 172.21.15.107:0/721557071 learned_addr learned my addr 172.21.15.107:0/721557071 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:12.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 client.?.objecter ms_handle_connect 0x7f4d4c0d11e0 2024-03-29T11:04:12.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:12.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670902287 0 0) 0x7f4d4c036550 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:12.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:12.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:12.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:04:12.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d38004eb0 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 client.?.objecter ms_handle_connect 0x7f4d4c0324b0 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320072021 0 0) 0x7f4d38004eb0 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d4c036550 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4d5404f400 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): dump: 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:12.178 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028040842 0 0) 0x7f4d4c160cd0 con 0x7f4d4c0324b0 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.193+0000 7f4d3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.194+0000 7f4d3ffff640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.194+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d38004eb0 con 0x7f4d4c0324b0 2024-03-29T11:04:12.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.194+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2623867133 0 0) 0x7f4d4c036550 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.194+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 >> v1:172.21.15.129:6789/0 conn(0x7f4d4c0324b0 legacy=0x7f4d4c15fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.194+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 >> v1:172.21.15.107:6789/0 conn(0x7f4d4c038c60 legacy=0x7f4d4c0bb1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.195+0000 7f4d3ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.195+0000 7f4d3ffff640 1 monclient: found mon.c 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.195+0000 7f4d3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.195+0000 7f4d3ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c160490 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 10 monclient: _finish_auth 0 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d5404f730 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d58dab640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d58dab640 10 monclient: _renew_subs 2024-03-29T11:04:12.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d58dab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c0be750 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d58dab640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f4d54067d90 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.196+0000 7f4d3ffff640 10 monclient: dump: 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:12.181 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.197+0000 7f4d58dab640 10 monclient: _renew_subs 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.197+0000 7f4d58dab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.197+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c0bb8f0 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.197+0000 7f4d58dab640 1 librados: init done 2024-03-29T11:04:12.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.197+0000 7f4d517fa640 4 set_mon_vals no callback set 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.198+0000 7f4d58dab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.198+0000 7f4d58dab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.198+0000 7f4d58dab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.198+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c007b70 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.199+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f4d54067ef0 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.200+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f4d540ff4f0 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.200+0000 7f4d3ffff640 10 client.4517.objecter ms_dispatch 0x7f4d54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.200+0000 7f4d3ffff640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.200+0000 7f4d537fe640 10 monclient: get_auth_request con 0x7f4d3806d950 auth_method 0 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.200+0000 7f4d3ffff640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.200+0000 7f4d3ffff640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-29T11:04:12.184 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.201+0000 7f4d3ffff640 10 client.4517.objecter ms_handle_connect 0x7f4d3806d950 2024-03-29T11:04:12.187 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.208+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d540cadd0 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.208+0000 7f4d3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:12.188 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.208+0000 7f4d3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:12.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.529+0000 7f4d58dab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:12.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.529+0000 7f4d58dab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:12.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.529+0000 7f4d58dab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:12.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.529+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d4c036a30 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.533+0000 7f4d3ffff640 1 -- 172.21.15.107:0/721557071 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f4d540ccd00 con 0x7f4d4c0d11e0 2024-03-29T11:04:12.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.533+0000 7f4d3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:12.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.533+0000 7f4d3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:12.513 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:04:12.513 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:12.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.536+0000 7f4d58dab640 10 librados: watch_flush enter 2024-03-29T11:04:12.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.536+0000 7f4d58dab640 10 librados: watch_flush exit 2024-03-29T11:04:12.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.536+0000 7f4d58dab640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-29T11:04:12.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.536+0000 7f4d58dab640 10 client.4517.objecter successfully canceled tick 2024-03-29T11:04:12.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.540+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 >> v1:172.21.15.107:6816/882194755 conn(0x7f4d3806d950 legacy=0x7f4d3806fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:12.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.540+0000 7f4d58dab640 10 monclient: shutdown 2024-03-29T11:04:12.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.540+0000 7f4d58dab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:12.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.540+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 >> v1:172.21.15.107:6790/0 conn(0x7f4d4c0d11e0 legacy=0x7f4d4c0343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:12.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.540+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 shutdown_connections 2024-03-29T11:04:12.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.540+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 >> 172.21.15.107:0/721557071 conn(0x7f4d4c005c20 msgr2=0x7f4d4c03ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:12.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.540+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 shutdown_connections 2024-03-29T11:04:12.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.540+0000 7f4d58dab640 1 -- 172.21.15.107:0/721557071 wait complete. 2024-03-29T11:04:12.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:12.540+0000 7f4d58dab640 1 librados: shutdown 2024-03-29T11:04:12.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:19.345 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:19.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.515+0000 7fd48b27b640 10 monclient: get_monmap_and_config 2024-03-29T11:04:19.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.515+0000 7fd48b27b640 10 monclient: build_initial_monmap 2024-03-29T11:04:19.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.515+0000 7fd48b27b640 10 monclient: monmap: 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:04:19.516779+0000 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:04:19.516779+0000 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.516+0000 7fd48b27b640 1 Processor -- start 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.516+0000 7fd48b27b640 1 -- start start 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.516+0000 7fd48b27b640 10 monclient: init 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.517+0000 7fd48b27b640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:19.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.517+0000 7fd48b27b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:04:19.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.517+0000 7fd48b27b640 10 monclient(hunting): picked mon.noname-a con 0x7fd47c029f00 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:19.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.518+0000 7fd48b27b640 10 monclient(hunting): picked mon.noname-b con 0x7fd47c02b090 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:19.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.518+0000 7fd48b27b640 10 monclient(hunting): picked mon.noname-c con 0x7fd47c0e2c40 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:19.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.518+0000 7fd48b27b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c02c0f0 con 0x7fd47c029f00 2024-03-29T11:04:19.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.518+0000 7fd48b27b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c02a430 con 0x7fd47c0e2c40 2024-03-29T11:04:19.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.518+0000 7fd48b27b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c02a5c0 con 0x7fd47c02b090 2024-03-29T11:04:19.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.518+0000 7fd48b27b640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:19.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.518+0000 7fd48b27b640 10 monclient(hunting): authenticate will time out at 1190.786039s 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.519+0000 7fd489a78640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd47c02b090 0x7fd47c0e2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46826/0 (socket says 172.21.15.107:46826) 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.519+0000 7fd489a78640 1 -- 172.21.15.107:0/3321510952 learned_addr learned my addr 172.21.15.107:0/3321510952 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.520+0000 7fd489277640 1 -- 172.21.15.107:0/3321510952 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395734031 0 0) 0x7fd47c02a5c0 con 0x7fd47c02b090 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.520+0000 7fd489277640 10 monclient(hunting): _negotiate 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.520+0000 7fd489277640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.520+0000 7fd489277640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.520+0000 7fd489277640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.520+0000 7fd489277640 1 -- 172.21.15.107:0/3321510952 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd470003500 con 0x7fd47c02b090 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.520+0000 7fd489277640 1 -- 172.21.15.107:0/3321510952 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 644186855 0 0) 0x7fd470003500 con 0x7fd47c02b090 2024-03-29T11:04:19.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.520+0000 7fd489277640 1 -- 172.21.15.107:0/3321510952 >> v1:172.21.15.129:6789/0 conn(0x7fd47c0e2c40 legacy=0x7fd47c0f0ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:19.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 1 -- 172.21.15.107:0/3321510952 >> v1:172.21.15.107:6789/0 conn(0x7fd47c029f00 legacy=0x7fd47c0e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:19.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 10 monclient: _finish_hunting 0 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 1 monclient: found mon.noname-b 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 1 -- 172.21.15.107:0/3321510952 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c006940 con 0x7fd47c02b090 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 10 monclient: _finish_auth 0 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 10 monclient: _check_auth_tickets 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 1 -- 172.21.15.107:0/3321510952 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fd4800012f0 con 0x7fd47c02b090 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 10 monclient: dump: 2024-03-29T11:04:19.505 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:19.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd48b27b640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:04:19.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd48b27b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:19.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 1 -- 172.21.15.107:0/3321510952 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd480001580 con 0x7fd47c02b090 2024-03-29T11:04:19.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd489277640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:19.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd48b27b640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:19.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.525+0000 7fd48b27b640 4 set_mon_vals no callback set 2024-03-29T11:04:19.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.526+0000 7fd489277640 1 -- 172.21.15.107:0/3321510952 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fd480001910 con 0x7fd47c02b090 2024-03-29T11:04:19.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.526+0000 7fd489277640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:19.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.526+0000 7fd489277640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:19.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.526+0000 7fd489277640 10 monclient: dump: 2024-03-29T11:04:19.508 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:19.508 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.526+0000 7fd48b27b640 10 monclient: shutdown 2024-03-29T11:04:19.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.526+0000 7fd48b27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:19.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.526+0000 7fd48b27b640 1 -- 172.21.15.107:0/3321510952 >> v1:172.21.15.107:6790/0 conn(0x7fd47c02b090 legacy=0x7fd47c0e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:19.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.527+0000 7fd48b27b640 1 -- 172.21.15.107:0/3321510952 shutdown_connections 2024-03-29T11:04:19.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.527+0000 7fd48b27b640 1 -- 172.21.15.107:0/3321510952 >> 172.21.15.107:0/3321510952 conn(0x7fd47c0058f0 msgr2=0x7fd47c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:19.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.528+0000 7fd48b27b640 1 -- 172.21.15.107:0/3321510952 shutdown_connections 2024-03-29T11:04:19.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.528+0000 7fd48b27b640 1 -- 172.21.15.107:0/3321510952 wait complete. 2024-03-29T11:04:19.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.529+0000 7fd48b27b640 10 monclient: build_initial_monmap 2024-03-29T11:04:19.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.529+0000 7fd48b27b640 10 monclient: monmap: 2024-03-29T11:04:19.512 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:19.512 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:19.512 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:04:19.512 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.529+0000 7fd48b27b640 1 librados: starting msgr at 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.529+0000 7fd48b27b640 1 librados: starting objecter 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.530+0000 7fd48b27b640 1 Processor -- start 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.530+0000 7fd48b27b640 1 -- start start 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.530+0000 7fd48b27b640 1 librados: setting wanted keys 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.530+0000 7fd48b27b640 1 librados: calling monclient init 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.530+0000 7fd48b27b640 10 monclient: init 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.530+0000 7fd48b27b640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:19.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.531+0000 7fd48b27b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:19.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.532+0000 7fd48b27b640 10 monclient(hunting): picked mon.noname-a con 0x7fd47c029f00 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:19.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.532+0000 7fd48b27b640 10 monclient(hunting): picked mon.noname-b con 0x7fd47c02b090 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:19.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.532+0000 7fd48b27b640 10 monclient(hunting): picked mon.noname-c con 0x7fd47c0e2c40 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:19.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.532+0000 7fd48b27b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c0de0b0 con 0x7fd47c029f00 2024-03-29T11:04:19.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.532+0000 7fd48b27b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c0d0440 con 0x7fd47c02b090 2024-03-29T11:04:19.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.532+0000 7fd48b27b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c0df760 con 0x7fd47c0e2c40 2024-03-29T11:04:19.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.532+0000 7fd48b27b640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:19.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.532+0000 7fd48b27b640 10 monclient(hunting): authenticate will time out at 1190.800471s 2024-03-29T11:04:19.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.535+0000 7fd48aa7a640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fd47c0e2c40 0x7fd47c185930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:49556/0 (socket says 172.21.15.107:49556) 2024-03-29T11:04:19.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.535+0000 7fd48aa7a640 1 -- 172.21.15.107:0/1296289494 learned_addr learned my addr 172.21.15.107:0/1296289494 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:19.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.535+0000 7fd47affd640 10 client.?.objecter ms_handle_connect 0x7fd47c0e2c40 2024-03-29T11:04:19.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.535+0000 7fd47affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:19.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.538+0000 7fd47affd640 10 client.?.objecter ms_handle_connect 0x7fd47c029f00 2024-03-29T11:04:19.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.538+0000 7fd47affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.538+0000 7fd47affd640 10 client.?.objecter ms_handle_connect 0x7fd47c02b090 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.538+0000 7fd47affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.538+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760559979 0 0) 0x7fd47c0de0b0 con 0x7fd47c029f00 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.538+0000 7fd47affd640 10 monclient(hunting): _negotiate 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.538+0000 7fd47affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd468004eb0 con 0x7fd47c029f00 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104541120 0 0) 0x7fd47c0d0440 con 0x7fd47c02b090 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): _negotiate 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:04:19.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47c0de0b0 con 0x7fd47c02b090 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266925201 0 0) 0x7fd47c0df760 con 0x7fd47c0e2c40 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): _negotiate 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47c0d0440 con 0x7fd47c0e2c40 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1595113263 0 0) 0x7fd468004eb0 con 0x7fd47c029f00 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd47c0df760 con 0x7fd47c029f00 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fd48404fdd0 con 0x7fd47c029f00 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): dump: 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:19.520 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.521 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.521 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:19.521 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:19.521 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:19.521 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:19.521 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:19.521 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:19.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:19.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1461283427 0 0) 0x7fd47c0d0440 con 0x7fd47c0e2c40 2024-03-29T11:04:19.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd468004eb0 con 0x7fd47c0e2c40 2024-03-29T11:04:19.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fd474001750 con 0x7fd47c0e2c40 2024-03-29T11:04:19.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:19.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:19.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.539+0000 7fd47affd640 10 monclient(hunting): dump: 2024-03-29T11:04:19.524 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:19.524 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1573591777 0 0) 0x7fd47c0df760 con 0x7fd47c029f00 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 >> v1:172.21.15.107:6790/0 conn(0x7fd47c0e2c40 legacy=0x7fd47c185930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 >> v1:172.21.15.129:6789/0 conn(0x7fd47c02b090 legacy=0x7fd47c0dbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 10 monclient: _finish_hunting 0 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 1 monclient: found mon.a 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:19.525 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c0d75c0 con 0x7fd47c029f00 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 10 monclient: _finish_auth 0 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 10 monclient: _check_auth_tickets 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd47affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd48b27b640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd48b27b640 10 monclient: _renew_subs 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd48b27b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd48b27b640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47c0de300 con 0x7fd47c029f00 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd48b27b640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd48b27b640 10 monclient: _renew_subs 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd48b27b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd48b27b640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47c0dde20 con 0x7fd47c029f00 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.540+0000 7fd48b27b640 1 librados: init done 2024-03-29T11:04:19.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.541+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd48404f400 con 0x7fd47c029f00 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.541+0000 7fd47affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.541+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fd484075f10 con 0x7fd47c029f00 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.541+0000 7fd47affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.541+0000 7fd47affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.541+0000 7fd47affd640 10 monclient: dump: 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:19.527 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:19.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.542+0000 7fd488a76640 4 set_mon_vals no callback set 2024-03-29T11:04:19.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.543+0000 7fd478ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:19.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.543+0000 7fd478ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:19.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.543+0000 7fd478ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:19.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.543+0000 7fd478ff9640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47c004cd0 con 0x7fd47c029f00 2024-03-29T11:04:19.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.544+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fd484067bf0 con 0x7fd47c029f00 2024-03-29T11:04:19.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.544+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fd4840ed190 con 0x7fd47c029f00 2024-03-29T11:04:19.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.544+0000 7fd489a78640 10 monclient: get_auth_request con 0x7fd46806dfa0 auth_method 0 2024-03-29T11:04:19.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.545+0000 7fd47affd640 10 client.4461.objecter ms_dispatch 0x7fd484000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:19.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.545+0000 7fd47affd640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:19.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.545+0000 7fd47affd640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:19.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.545+0000 7fd47affd640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:04:19.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.545+0000 7fd47affd640 10 client.4461.objecter ms_handle_connect 0x7fd46806dfa0 2024-03-29T11:04:19.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.553+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4840b8a70 con 0x7fd47c029f00 2024-03-29T11:04:19.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.553+0000 7fd47affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:19.532 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.553+0000 7fd47affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:19.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.875+0000 7fd478ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:19.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.875+0000 7fd478ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:19.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.875+0000 7fd478ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:19.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.875+0000 7fd478ff9640 1 -- 172.21.15.107:0/1296289494 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd47c0de490 con 0x7fd47c029f00 2024-03-29T11:04:19.858 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:04:19.858 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:19.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.876+0000 7fd47affd640 1 -- 172.21.15.107:0/1296289494 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fd4840ba9a0 con 0x7fd47c029f00 2024-03-29T11:04:19.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.876+0000 7fd47affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:19.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.876+0000 7fd47affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:19.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.879+0000 7fd478ff9640 10 librados: watch_flush enter 2024-03-29T11:04:19.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.879+0000 7fd478ff9640 10 librados: watch_flush exit 2024-03-29T11:04:19.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.879+0000 7fd478ff9640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:04:19.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.879+0000 7fd478ff9640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:04:19.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.879+0000 7fd478ff9640 1 -- 172.21.15.107:0/1296289494 >> v1:172.21.15.107:6816/882194755 conn(0x7fd46806dfa0 legacy=0x7fd468070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:19.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.879+0000 7fd478ff9640 10 monclient: shutdown 2024-03-29T11:04:19.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.879+0000 7fd478ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:19.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.879+0000 7fd478ff9640 1 -- 172.21.15.107:0/1296289494 >> v1:172.21.15.107:6789/0 conn(0x7fd47c029f00 legacy=0x7fd47c0cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:19.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.879+0000 7fd478ff9640 1 -- 172.21.15.107:0/1296289494 shutdown_connections 2024-03-29T11:04:19.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.880+0000 7fd478ff9640 1 -- 172.21.15.107:0/1296289494 >> 172.21.15.107:0/1296289494 conn(0x7fd47c0058f0 msgr2=0x7fd47c0e2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:19.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.880+0000 7fd478ff9640 1 -- 172.21.15.107:0/1296289494 shutdown_connections 2024-03-29T11:04:19.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.880+0000 7fd478ff9640 1 -- 172.21.15.107:0/1296289494 wait complete. 2024-03-29T11:04:19.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:19.880+0000 7fd478ff9640 1 librados: shutdown 2024-03-29T11:04:19.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:26.681 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.850+0000 7f3ed5e25640 10 monclient: get_monmap_and_config 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.850+0000 7f3ed5e25640 10 monclient: build_initial_monmap 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.850+0000 7f3ed5e25640 10 monclient: monmap: 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:04:26.851429+0000 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:04:26.851429+0000 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:26.830 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:04:26.831 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:04:26.831 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:26.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.851+0000 7f3ed5e25640 1 Processor -- start 2024-03-29T11:04:26.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.851+0000 7f3ed5e25640 1 -- start start 2024-03-29T11:04:26.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.851+0000 7f3ed5e25640 10 monclient: init 2024-03-29T11:04:26.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.852+0000 7f3ed5e25640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:26.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.852+0000 7f3ed5e25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:04:26.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.852+0000 7f3ed5e25640 10 monclient(hunting): picked mon.noname-a con 0x7f3ed00b0860 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:26.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ed5e25640 10 monclient(hunting): picked mon.noname-b con 0x7f3ed013ef30 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:26.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ecf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3ed00b0860 0x7f3ed013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33232/0 (socket says 172.21.15.107:33232) 2024-03-29T11:04:26.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ed5e25640 10 monclient(hunting): picked mon.noname-c con 0x7f3ed00af8e0 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:26.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ecf7fe640 1 -- 172.21.15.107:0/890361575 learned_addr learned my addr 172.21.15.107:0/890361575 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:26.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ed5e25640 1 -- 172.21.15.107:0/890361575 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed00b0c40 con 0x7f3ed00b0860 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ed5e25640 1 -- 172.21.15.107:0/890361575 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed00aed10 con 0x7f3ed00af8e0 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ed5e25640 1 -- 172.21.15.107:0/890361575 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed00afd50 con 0x7f3ed013ef30 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ed5e25640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ed5e25640 10 monclient(hunting): authenticate will time out at 1198.121808s 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.853+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553713001 0 0) 0x7f3ed00afd50 con 0x7f3ed013ef30 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.854+0000 7f3ece7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.854+0000 7f3ece7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.854+0000 7f3ece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.854+0000 7f3ece7fc640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.854+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb8003520 con 0x7f3ed013ef30 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.854+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74257692 0 0) 0x7f3ed00aed10 con 0x7f3ed00af8e0 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.854+0000 7f3ece7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.854+0000 7f3ece7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:26.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:04:26.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed00afd50 con 0x7f3ed00af8e0 2024-03-29T11:04:26.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257610588 0 0) 0x7f3ed00b0c40 con 0x7f3ed00b0860 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed00aed10 con 0x7f3ed00b0860 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2788098592 0 0) 0x7f3eb8003520 con 0x7f3ed013ef30 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 >> v1:172.21.15.129:6789/0 conn(0x7f3ed00af8e0 legacy=0x7f3ed0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 >> v1:172.21.15.107:6789/0 conn(0x7f3ed00b0860 legacy=0x7f3ed013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:04:26.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 1 monclient: found mon.noname-b 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed00ab1b0 con 0x7f3ed013ef30 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient: _finish_auth 0 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3ec4001610 con 0x7f3ed013ef30 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ece7fc640 10 monclient: dump: 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.841 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ed5e25640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.855+0000 7f3ed5e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.857+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ec4001770 con 0x7f3ed013ef30 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.857+0000 7f3ece7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.857+0000 7f3ece7fc640 1 -- 172.21.15.107:0/890361575 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3ec4001bc0 con 0x7f3ed013ef30 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.857+0000 7f3ece7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.857+0000 7f3ece7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.857+0000 7f3ece7fc640 10 monclient: dump: 2024-03-29T11:04:26.842 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.857+0000 7f3ed5e25640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.857+0000 7f3ed5e25640 4 set_mon_vals no callback set 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.858+0000 7f3ed5e25640 10 monclient: shutdown 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.858+0000 7f3ed5e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.858+0000 7f3ed5e25640 1 -- 172.21.15.107:0/890361575 >> v1:172.21.15.107:6790/0 conn(0x7f3ed013ef30 legacy=0x7f3ed00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:26.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.858+0000 7f3ed5e25640 1 -- 172.21.15.107:0/890361575 shutdown_connections 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.858+0000 7f3ed5e25640 1 -- 172.21.15.107:0/890361575 >> 172.21.15.107:0/890361575 conn(0x7f3ed006b660 msgr2=0x7f3ed006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.858+0000 7f3ed5e25640 1 -- 172.21.15.107:0/890361575 shutdown_connections 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.858+0000 7f3ed5e25640 1 -- 172.21.15.107:0/890361575 wait complete. 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.859+0000 7f3ed5e25640 10 monclient: build_initial_monmap 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.859+0000 7f3ed5e25640 10 monclient: monmap: 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:04:26.844 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:26.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.860+0000 7f3ed5e25640 1 librados: starting msgr at 2024-03-29T11:04:26.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.860+0000 7f3ed5e25640 1 librados: starting objecter 2024-03-29T11:04:26.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.860+0000 7f3ed5e25640 1 Processor -- start 2024-03-29T11:04:26.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.860+0000 7f3ed5e25640 1 -- start start 2024-03-29T11:04:26.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.860+0000 7f3ed5e25640 1 librados: setting wanted keys 2024-03-29T11:04:26.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.860+0000 7f3ed5e25640 1 librados: calling monclient init 2024-03-29T11:04:26.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.860+0000 7f3ed5e25640 10 monclient: init 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 10 monclient(hunting): picked mon.noname-a con 0x7f3ed00af8e0 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 10 monclient(hunting): picked mon.noname-b con 0x7f3ed00b0860 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 10 monclient(hunting): picked mon.noname-c con 0x7f3ed013ef30 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed01d66d0 con 0x7f3ed00af8e0 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed00b3d10 con 0x7f3ed00b0860 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed0139e80 con 0x7f3ed013ef30 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.861+0000 7f3ed5e25640 10 monclient(hunting): authenticate will time out at 1198.129810s 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.862+0000 7f3ecffff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3ed013ef30 0x7f3ed01ddbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54696/0 (socket says 172.21.15.107:54696) 2024-03-29T11:04:26.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.862+0000 7f3ecffff640 1 -- 172.21.15.107:0/2655882369 learned_addr learned my addr 172.21.15.107:0/2655882369 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.862+0000 7f3eb7fff640 10 client.?.objecter ms_handle_connect 0x7f3ed013ef30 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.862+0000 7f3eb7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.862+0000 7f3eb7fff640 10 client.?.objecter ms_handle_connect 0x7f3ed00af8e0 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.862+0000 7f3eb7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.862+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081631334 0 0) 0x7f3ed0139e80 con 0x7f3ed013ef30 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.862+0000 7f3eb7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.862+0000 7f3eb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc004eb0 con 0x7f3ed013ef30 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 client.?.objecter ms_handle_connect 0x7f3ed00b0860 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011689229 0 0) 0x7f3ed01d66d0 con 0x7f3ed00af8e0 2024-03-29T11:04:26.852 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed0139e80 con 0x7f3ed00af8e0 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3451580876 0 0) 0x7f3ed0139e80 con 0x7f3ed00af8e0 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed01d66d0 con 0x7f3ed00af8e0 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3ec0001870 con 0x7f3ed00af8e0 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.863+0000 7f3eb7fff640 10 monclient(hunting): dump: 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.853 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221151549 0 0) 0x7f3ed00b3d10 con 0x7f3ed00b0860 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed0139e80 con 0x7f3ed00b0860 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1009355233 0 0) 0x7f3ebc004eb0 con 0x7f3ed013ef30 2024-03-29T11:04:26.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed00b3d10 con 0x7f3ed013ef30 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3ec8001e10 con 0x7f3ed013ef30 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient(hunting): dump: 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:26.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2268626789 0 0) 0x7f3ed01d66d0 con 0x7f3ed00af8e0 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 >> v1:172.21.15.107:6790/0 conn(0x7f3ed013ef30 legacy=0x7f3ed01ddbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 >> v1:172.21.15.129:6789/0 conn(0x7f3ed00b0860 legacy=0x7f3ed00b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 monclient: found mon.a 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed013a010 con 0x7f3ed00af8e0 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient: _finish_auth 0 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3eb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3ed5e25640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3ed5e25640 10 monclient: _renew_subs 2024-03-29T11:04:26.856 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3ed5e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed00b1920 con 0x7f3ed00af8e0 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3ed5e25640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3ed5e25640 10 monclient: _renew_subs 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3ed5e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed01d7450 con 0x7f3ed00af8e0 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.864+0000 7f3ed5e25640 1 librados: init done 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.865+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ec0001c80 con 0x7f3ed00af8e0 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.865+0000 7f3eb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.865+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3ec0004980 con 0x7f3ed00af8e0 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.865+0000 7f3ecdffb640 4 set_mon_vals no callback set 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.865+0000 7f3eb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.865+0000 7f3eb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:26.857 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.865+0000 7f3eb7fff640 10 monclient: dump: 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.866+0000 7f3ed5e25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.866+0000 7f3ed5e25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.866+0000 7f3ed5e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:26.858 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.866+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb0005080 con 0x7f3ed00af8e0 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.867+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f3ec0004ae0 con 0x7f3ed00af8e0 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.868+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f3ec0084930 con 0x7f3ed00af8e0 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.868+0000 7f3eb7fff640 10 client.4467.objecter ms_dispatch 0x7f3ed0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.868+0000 7f3eb7fff640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.868+0000 7f3eb7fff640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.869+0000 7f3eb7fff640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.869+0000 7f3eceffd640 10 monclient: get_auth_request con 0x7f3ebc06e020 auth_method 0 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.869+0000 7f3eb7fff640 10 client.4467.objecter ms_handle_connect 0x7f3ebc06e020 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.875+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ec0050210 con 0x7f3ed00af8e0 2024-03-29T11:04:26.859 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.875+0000 7f3eb7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:26.860 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:26.875+0000 7f3eb7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:27.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.236+0000 7f3ed5e25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:27.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.236+0000 7f3ed5e25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:27.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.236+0000 7f3ed5e25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:27.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.236+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3eb0005330 con 0x7f3ed00af8e0 2024-03-29T11:04:27.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.238+0000 7f3eb7fff640 1 -- 172.21.15.107:0/2655882369 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f3ec0052140 con 0x7f3ed00af8e0 2024-03-29T11:04:27.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.238+0000 7f3eb7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:27.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.238+0000 7f3eb7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:27.217 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:04:27.218 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:27.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.242+0000 7f3ed5e25640 10 librados: watch_flush enter 2024-03-29T11:04:27.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.242+0000 7f3ed5e25640 10 librados: watch_flush exit 2024-03-29T11:04:27.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.242+0000 7f3ed5e25640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T11:04:27.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.242+0000 7f3ed5e25640 10 client.4467.objecter successfully canceled tick 2024-03-29T11:04:27.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.242+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 >> v1:172.21.15.107:6816/882194755 conn(0x7f3ebc06e020 legacy=0x7f3ebc0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.243+0000 7f3ed5e25640 10 monclient: shutdown 2024-03-29T11:04:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.243+0000 7f3ed5e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.243+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 >> v1:172.21.15.107:6789/0 conn(0x7f3ed00af8e0 legacy=0x7f3ed01d6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.243+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 shutdown_connections 2024-03-29T11:04:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.243+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 >> 172.21.15.107:0/2655882369 conn(0x7f3ed006b660 msgr2=0x7f3ed013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.243+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 shutdown_connections 2024-03-29T11:04:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.243+0000 7f3ed5e25640 1 -- 172.21.15.107:0/2655882369 wait complete. 2024-03-29T11:04:27.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:27.243+0000 7f3ed5e25640 1 librados: shutdown 2024-03-29T11:04:27.238 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:34.045 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:34.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.228+0000 7f309e24a640 10 monclient: get_monmap_and_config 2024-03-29T11:04:34.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.228+0000 7f309e24a640 10 monclient: build_initial_monmap 2024-03-29T11:04:34.208 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.228+0000 7f309e24a640 10 monclient: monmap: 2024-03-29T11:04:34.208 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:34.209 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:34.209 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:04:34.229775+0000 2024-03-29T11:04:34.209 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:04:34.229775+0000 2024-03-29T11:04:34.209 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:34.209 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:34.209 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:34.209 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:04:34.209 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:04:34.210 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:34.210 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.229+0000 7f309e24a640 1 Processor -- start 2024-03-29T11:04:34.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 1 -- start start 2024-03-29T11:04:34.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 10 monclient: init 2024-03-29T11:04:34.220 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 10 monclient(hunting): picked mon.noname-c con 0x7f30980a9750 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 10 monclient(hunting): picked mon.noname-b con 0x7f3098149890 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 10 monclient(hunting): picked mon.noname-a con 0x7f30980afa30 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30980b00c0 con 0x7f30980afa30 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30981464a0 con 0x7f30980a9750 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30980ab280 con 0x7f3098149890 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.230+0000 7f309e24a640 10 monclient(hunting): authenticate will time out at 1205.498919s 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.233+0000 7f3096ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3098149890 0x7f3098149da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51584/0 (socket says 172.21.15.107:51584) 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.233+0000 7f3096ffd640 1 -- 172.21.15.107:0/2185760342 learned_addr learned my addr 172.21.15.107:0/2185760342 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092996867 0 0) 0x7f30980ab280 con 0x7f3098149890 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:34.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3080003520 con 0x7f3098149890 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846165877 0 0) 0x7f30981464a0 con 0x7f30980a9750 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.235+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30980ab280 con 0x7f30980a9750 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 259227320 0 0) 0x7f3080003520 con 0x7f3098149890 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 >> v1:172.21.15.129:6789/0 conn(0x7f30980a9750 legacy=0x7f30981469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 >> v1:172.21.15.107:6789/0 conn(0x7f30980afa30 legacy=0x7f3098151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:04:34.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 1 monclient: found mon.noname-b 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30981470b0 con 0x7f3098149890 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 10 monclient: _finish_auth 0 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f309e24a640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f309e24a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3090001a10 con 0x7f3098149890 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:34.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 10 monclient: dump: 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f309e24a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 1 -- 172.21.15.107:0/2185760342 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3090001b70 con 0x7f3098149890 2024-03-29T11:04:34.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f30967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f309e24a640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.236+0000 7f309e24a640 4 set_mon_vals no callback set 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.237+0000 7f309e24a640 10 monclient: shutdown 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.237+0000 7f309e24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.237+0000 7f309e24a640 1 -- 172.21.15.107:0/2185760342 >> v1:172.21.15.107:6790/0 conn(0x7f3098149890 legacy=0x7f3098149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.238+0000 7f309e24a640 1 -- 172.21.15.107:0/2185760342 shutdown_connections 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.238+0000 7f309e24a640 1 -- 172.21.15.107:0/2185760342 >> 172.21.15.107:0/2185760342 conn(0x7f30980b3a00 msgr2=0x7f30980b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.238+0000 7f309e24a640 1 -- 172.21.15.107:0/2185760342 shutdown_connections 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.239+0000 7f309e24a640 1 -- 172.21.15.107:0/2185760342 wait complete. 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.240+0000 7f309e24a640 10 monclient: build_initial_monmap 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.240+0000 7f309e24a640 10 monclient: monmap: 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:34.225 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.240+0000 7f309e24a640 1 librados: starting msgr at 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.240+0000 7f309e24a640 1 librados: starting objecter 2024-03-29T11:04:34.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.240+0000 7f309e24a640 1 Processor -- start 2024-03-29T11:04:34.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 1 -- start start 2024-03-29T11:04:34.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 1 librados: setting wanted keys 2024-03-29T11:04:34.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 1 librados: calling monclient init 2024-03-29T11:04:34.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 10 monclient: init 2024-03-29T11:04:34.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:34.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:04:34.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 10 monclient(hunting): picked mon.noname-a con 0x7f30980a9750 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:34.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 10 monclient(hunting): picked mon.noname-c con 0x7f30980afa30 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:34.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 10 monclient(hunting): picked mon.noname-b con 0x7f3098149890 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:34.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3098142840 con 0x7f30980a9750 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309806e300 con 0x7f3098149890 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3098142cd0 con 0x7f30980afa30 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.241+0000 7f309e24a640 10 monclient(hunting): authenticate will time out at 1205.509831s 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.242+0000 7f3096ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f30980afa30 0x7f309813c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55824/0 (socket says 172.21.15.107:55824) 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.242+0000 7f3096ffd640 1 -- 172.21.15.107:0/2509976239 learned_addr learned my addr 172.21.15.107:0/2509976239 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.243+0000 7f307ffff640 10 client.?.objecter ms_handle_connect 0x7f30980afa30 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.243+0000 7f307ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.243+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495109190 0 0) 0x7f3098142cd0 con 0x7f30980afa30 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.243+0000 7f307ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.243+0000 7f307ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:34.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.243+0000 7f307ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.243+0000 7f307ffff640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.243+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3084004eb0 con 0x7f30980afa30 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.245+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4047813402 0 0) 0x7f3084004eb0 con 0x7f30980afa30 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.245+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3098142cd0 con 0x7f30980afa30 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.245+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3090004d80 con 0x7f30980afa30 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.245+0000 7f307ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.245+0000 7f307ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.245+0000 7f307ffff640 10 monclient(hunting): dump: 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:34.241 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.245+0000 7f307ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 10 client.?.objecter ms_handle_connect 0x7f3098149890 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 10 client.?.objecter ms_handle_connect 0x7f30980a9750 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3102380999 0 0) 0x7f3098142cd0 con 0x7f30980afa30 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 >> v1:172.21.15.129:6789/0 conn(0x7f3098149890 legacy=0x7f30981e6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 >> v1:172.21.15.107:6789/0 conn(0x7f30980a9750 legacy=0x7f30981db300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:04:34.242 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 1 monclient: found mon.c 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3098142e60 con 0x7f30980afa30 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 10 monclient: _finish_auth 0 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f309e24a640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f309e24a640 10 monclient: _renew_subs 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f309e24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30981456a0 con 0x7f30980afa30 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f309e24a640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f309e24a640 10 monclient: _renew_subs 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f309e24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30981dbbf0 con 0x7f30980afa30 2024-03-29T11:04:34.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f309e24a640 1 librados: init done 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3090001de0 con 0x7f30980afa30 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f307ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.246+0000 7f3095ffb640 4 set_mon_vals no callback set 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.247+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f30900053f0 con 0x7f30980afa30 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.247+0000 7f307ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.247+0000 7f307ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.247+0000 7f307ffff640 10 monclient: dump: 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:34.244 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.248+0000 7f309e24a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.248+0000 7f309e24a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.248+0000 7f309e24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.248+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3078005080 con 0x7f30980afa30 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.249+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f3090005550 con 0x7f30980afa30 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.250+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f3090085040 con 0x7f30980afa30 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.250+0000 7f307ffff640 10 client.4535.objecter ms_dispatch 0x7f3098000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.250+0000 7f307ffff640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.250+0000 7f307ffff640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.250+0000 7f307ffff640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.250+0000 7f30977fe640 10 monclient: get_auth_request con 0x7f308406db60 auth_method 0 2024-03-29T11:04:34.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.251+0000 7f307ffff640 10 client.4535.objecter ms_handle_connect 0x7f308406db60 2024-03-29T11:04:34.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.258+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3090050920 con 0x7f30980afa30 2024-03-29T11:04:34.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.258+0000 7f307ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:34.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.258+0000 7f307ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:34.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.580+0000 7f309e24a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:34.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.580+0000 7f309e24a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:34.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.580+0000 7f309e24a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:34.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.580+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3078005330 con 0x7f30980afa30 2024-03-29T11:04:34.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.580+0000 7f307ffff640 1 -- 172.21.15.107:0/2509976239 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f3090052850 con 0x7f30980afa30 2024-03-29T11:04:34.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.580+0000 7f307ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:34.560 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.580+0000 7f307ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:34.561 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:04:34.561 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.584+0000 7f309e24a640 10 librados: watch_flush enter 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.584+0000 7f309e24a640 10 librados: watch_flush exit 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.584+0000 7f309e24a640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.584+0000 7f309e24a640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.584+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 >> v1:172.21.15.107:6816/882194755 conn(0x7f308406db60 legacy=0x7f3084070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.584+0000 7f309e24a640 10 monclient: shutdown 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.584+0000 7f309e24a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.584+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 >> v1:172.21.15.107:6790/0 conn(0x7f30980afa30 legacy=0x7f309813c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.585+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 shutdown_connections 2024-03-29T11:04:34.565 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.585+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 >> 172.21.15.107:0/2509976239 conn(0x7f30980b3a00 msgr2=0x7f30980ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:34.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.585+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 shutdown_connections 2024-03-29T11:04:34.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.585+0000 7f309e24a640 1 -- 172.21.15.107:0/2509976239 wait complete. 2024-03-29T11:04:34.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:34.585+0000 7f309e24a640 1 librados: shutdown 2024-03-29T11:04:34.580 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:41.387 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:41.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.582+0000 7f9f38c63640 10 monclient: get_monmap_and_config 2024-03-29T11:04:41.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.582+0000 7f9f38c63640 10 monclient: build_initial_monmap 2024-03-29T11:04:41.566 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.582+0000 7f9f38c63640 10 monclient: monmap: 2024-03-29T11:04:41.566 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:41.566 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:41.566 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:04:41.583277+0000 2024-03-29T11:04:41.566 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:04:41.583277+0000 2024-03-29T11:04:41.566 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.583+0000 7f9f38c63640 1 Processor -- start 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.583+0000 7f9f38c63640 1 -- start start 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.583+0000 7f9f38c63640 10 monclient: init 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 10 monclient(hunting): picked mon.noname-b con 0x7f9f340ad6c0 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 10 monclient(hunting): picked mon.noname-a con 0x7f9f340af6c0 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 10 monclient(hunting): picked mon.noname-c con 0x7f9f3414fdc0 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340a94e0 con 0x7f9f340af6c0 2024-03-29T11:04:41.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340af3a0 con 0x7f9f3414fdc0 2024-03-29T11:04:41.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340afb60 con 0x7f9f340ad6c0 2024-03-29T11:04:41.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:41.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.584+0000 7f9f38c63640 10 monclient(hunting): authenticate will time out at 1212.852260s 2024-03-29T11:04:41.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.587+0000 7f9f3259c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9f340ad6c0 0x7f9f3414f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:34886/0 (socket says 172.21.15.107:34886) 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.587+0000 7f9f3259c640 1 -- 172.21.15.107:0/2052741781 learned_addr learned my addr 172.21.15.107:0/2052741781 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082737212 0 0) 0x7f9f340a94e0 con 0x7f9f340af6c0 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f14003520 con 0x7f9f340af6c0 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500282874 0 0) 0x7f9f340af3a0 con 0x7f9f3414fdc0 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f14003840 con 0x7f9f3414fdc0 2024-03-29T11:04:41.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945965384 0 0) 0x7f9f340afb60 con 0x7f9f340ad6c0 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f14003b80 con 0x7f9f340ad6c0 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 399504979 0 0) 0x7f9f14003520 con 0x7f9f340af6c0 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 >> v1:172.21.15.107:6790/0 conn(0x7f9f340ad6c0 legacy=0x7f9f3414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 >> v1:172.21.15.129:6789/0 conn(0x7f9f3414fdc0 legacy=0x7f9f3415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 monclient: found mon.noname-a 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:04:41.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f340aab80 con 0x7f9f340af6c0 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient: _finish_auth 0 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9f1c001670 con 0x7f9f340af6c0 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient: dump: 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:41.571 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:41.572 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:41.572 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:41.572 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:41.572 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:41.572 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:41.572 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:41.572 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:41.572 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f38c63640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:04:41.572 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f38c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f1c001850 con 0x7f9f340af6c0 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f31d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f38c63640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.589+0000 7f9f38c63640 4 set_mon_vals no callback set 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.590+0000 7f9f31d9b640 1 -- 172.21.15.107:0/2052741781 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9f1c001670 con 0x7f9f340af6c0 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.590+0000 7f9f31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.590+0000 7f9f31d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.590+0000 7f9f31d9b640 10 monclient: dump: 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:41.573 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.590+0000 7f9f38c63640 10 monclient: shutdown 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.590+0000 7f9f38c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.590+0000 7f9f38c63640 1 -- 172.21.15.107:0/2052741781 >> v1:172.21.15.107:6789/0 conn(0x7f9f340af6c0 legacy=0x7f9f34152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.590+0000 7f9f38c63640 1 -- 172.21.15.107:0/2052741781 shutdown_connections 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.590+0000 7f9f38c63640 1 -- 172.21.15.107:0/2052741781 >> 172.21.15.107:0/2052741781 conn(0x7f9f340b35e0 msgr2=0x7f9f340b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.592+0000 7f9f38c63640 1 -- 172.21.15.107:0/2052741781 shutdown_connections 2024-03-29T11:04:41.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.592+0000 7f9f38c63640 1 -- 172.21.15.107:0/2052741781 wait complete. 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.593+0000 7f9f38c63640 10 monclient: build_initial_monmap 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.593+0000 7f9f38c63640 10 monclient: monmap: 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:41.575 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:04:41.576 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:04:41.576 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:41.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.594+0000 7f9f38c63640 1 librados: starting msgr at 2024-03-29T11:04:41.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.594+0000 7f9f38c63640 1 librados: starting objecter 2024-03-29T11:04:41.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.594+0000 7f9f38c63640 1 Processor -- start 2024-03-29T11:04:41.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.595+0000 7f9f38c63640 1 -- start start 2024-03-29T11:04:41.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.595+0000 7f9f38c63640 1 librados: setting wanted keys 2024-03-29T11:04:41.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.595+0000 7f9f38c63640 1 librados: calling monclient init 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.595+0000 7f9f38c63640 10 monclient: init 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 10 monclient(hunting): picked mon.noname-a con 0x7f9f340ad6c0 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3414fdc0 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 10 monclient(hunting): picked mon.noname-c con 0x7f9f34071dc0 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f34071750 con 0x7f9f340ad6c0 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3413ddc0 con 0x7f9f3414fdc0 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340721f0 con 0x7f9f34071dc0 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:41.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.596+0000 7f9f38c63640 10 monclient(hunting): authenticate will time out at 1212.864760s 2024-03-29T11:04:41.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.599+0000 7f9f32d9d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9f34071dc0 0x7f9f3414a5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35686/0 (socket says 172.21.15.107:35686) 2024-03-29T11:04:41.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.599+0000 7f9f32d9d640 1 -- 172.21.15.107:0/698285999 learned_addr learned my addr 172.21.15.107:0/698285999 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:41.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.599+0000 7f9f2b7fe640 10 client.?.objecter ms_handle_connect 0x7f9f34071dc0 2024-03-29T11:04:41.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.599+0000 7f9f2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:41.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057549382 0 0) 0x7f9f340721f0 con 0x7f9f34071dc0 2024-03-29T11:04:41.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:41.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:41.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:41.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:04:41.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f18004ee0 con 0x7f9f34071dc0 2024-03-29T11:04:41.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 client.?.objecter ms_handle_connect 0x7f9f3414fdc0 2024-03-29T11:04:41.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:41.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 612370313 0 0) 0x7f9f18004ee0 con 0x7f9f34071dc0 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f340721f0 con 0x7f9f34071dc0 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9f2c001bf0 con 0x7f9f34071dc0 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 monclient(hunting): dump: 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:41.586 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.602+0000 7f9f2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905329000 0 0) 0x7f9f3413ddc0 con 0x7f9f3414fdc0 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f18004ee0 con 0x7f9f3414fdc0 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 client.?.objecter ms_handle_connect 0x7f9f340ad6c0 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 259348413 0 0) 0x7f9f340721f0 con 0x7f9f34071dc0 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 >> v1:172.21.15.129:6789/0 conn(0x7f9f3414fdc0 legacy=0x7f9f341e48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:41.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 >> v1:172.21.15.107:6789/0 conn(0x7f9f340ad6c0 legacy=0x7f9f34071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 1 monclient: found mon.c 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f341e4380 con 0x7f9f34071dc0 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f38c63640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f38c63640 10 monclient: _renew_subs 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f38c63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3406e7a0 con 0x7f9f34071dc0 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f38c63640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f38c63640 10 monclient: _renew_subs 2024-03-29T11:04:41.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f38c63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f341e5190 con 0x7f9f34071dc0 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.603+0000 7f9f38c63640 1 librados: init done 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.604+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f2c003970 con 0x7f9f34071dc0 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.604+0000 7f9f2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.604+0000 7f9f3159a640 4 set_mon_vals no callback set 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.605+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9f2c001d50 con 0x7f9f34071dc0 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.605+0000 7f9f2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.605+0000 7f9f2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.605+0000 7f9f2b7fe640 10 monclient: dump: 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:41.589 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:41.590 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:41.590 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:41.590 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:41.590 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:41.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.605+0000 7f9f38c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:41.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.605+0000 7f9f38c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:41.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.605+0000 7f9f38c63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:41.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.605+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0c005080 con 0x7f9f34071dc0 2024-03-29T11:04:41.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.608+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f9f2c003d80 con 0x7f9f34071dc0 2024-03-29T11:04:41.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.608+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f9f2c084cc0 con 0x7f9f34071dc0 2024-03-29T11:04:41.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.609+0000 7f9f2b7fe640 10 client.4541.objecter ms_dispatch 0x7f9f34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:41.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.609+0000 7f9f2b7fe640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:41.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.609+0000 7f9f3259c640 10 monclient: get_auth_request con 0x7f9f1806dd10 auth_method 0 2024-03-29T11:04:41.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.609+0000 7f9f2b7fe640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:41.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.609+0000 7f9f2b7fe640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:04:41.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.609+0000 7f9f2b7fe640 10 client.4541.objecter ms_handle_connect 0x7f9f1806dd10 2024-03-29T11:04:41.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.616+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f2c0505a0 con 0x7f9f34071dc0 2024-03-29T11:04:41.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.616+0000 7f9f2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:41.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.616+0000 7f9f2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:41.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.959+0000 7f9f38c63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:41.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.959+0000 7f9f38c63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:41.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.959+0000 7f9f38c63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:41.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.959+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f0c005330 con 0x7f9f34071dc0 2024-03-29T11:04:41.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.960+0000 7f9f2b7fe640 1 -- 172.21.15.107:0/698285999 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f9f2c0524d0 con 0x7f9f34071dc0 2024-03-29T11:04:41.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.960+0000 7f9f2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:41.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.960+0000 7f9f2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:41.942 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:04:41.943 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:41.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.966+0000 7f9f38c63640 10 librados: watch_flush enter 2024-03-29T11:04:41.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.966+0000 7f9f38c63640 10 librados: watch_flush exit 2024-03-29T11:04:41.946 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.966+0000 7f9f38c63640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:04:41.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.966+0000 7f9f38c63640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:04:41.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.967+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 >> v1:172.21.15.107:6816/882194755 conn(0x7f9f1806dd10 legacy=0x7f9f180701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:41.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.967+0000 7f9f38c63640 10 monclient: shutdown 2024-03-29T11:04:41.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.967+0000 7f9f38c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:41.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.967+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 >> v1:172.21.15.107:6790/0 conn(0x7f9f34071dc0 legacy=0x7f9f3414a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:41.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.967+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 shutdown_connections 2024-03-29T11:04:41.947 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.967+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 >> 172.21.15.107:0/698285999 conn(0x7f9f340b35e0 msgr2=0x7f9f3414f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.968+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 shutdown_connections 2024-03-29T11:04:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.968+0000 7f9f38c63640 1 -- 172.21.15.107:0/698285999 wait complete. 2024-03-29T11:04:41.948 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:41.969+0000 7f9f38c63640 1 librados: shutdown 2024-03-29T11:04:41.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:48.771 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:48.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.927+0000 7fa4fd77b640 10 monclient: get_monmap_and_config 2024-03-29T11:04:48.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.927+0000 7fa4fd77b640 10 monclient: build_initial_monmap 2024-03-29T11:04:48.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.927+0000 7fa4fd77b640 10 monclient: monmap: 2024-03-29T11:04:48.907 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:48.907 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:48.907 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:04:48.928830+0000 2024-03-29T11:04:48.907 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:04:48.928830+0000 2024-03-29T11:04:48.907 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:48.908 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:48.908 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:48.908 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:04:48.908 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:04:48.908 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:48.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.928+0000 7fa4fd77b640 1 Processor -- start 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.928+0000 7fa4fd77b640 1 -- start start 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.928+0000 7fa4fd77b640 10 monclient: init 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.929+0000 7fa4fd77b640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.929+0000 7fa4fd77b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.929+0000 7fa4f6ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa4f81457f0 0x7fa4f813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53994/0 (socket says 172.21.15.107:53994) 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.929+0000 7fa4f6ffd640 1 -- 172.21.15.107:0/3599819693 learned_addr learned my addr 172.21.15.107:0/3599819693 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.929+0000 7fa4fd77b640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f81457f0 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.929+0000 7fa4fd77b640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f813f650 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.929+0000 7fa4fd77b640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f80a9d70 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4fd77b640 1 -- 172.21.15.107:0/3599819693 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f80aaef0 con 0x7fa4f813f650 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4fd77b640 1 -- 172.21.15.107:0/3599819693 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f8145bd0 con 0x7fa4f80a9d70 2024-03-29T11:04:48.910 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4fd77b640 1 -- 172.21.15.107:0/3599819693 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f8145eb0 con 0x7fa4f81457f0 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4fd77b640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4fd77b640 10 monclient(hunting): authenticate will time out at 1220.198162s 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539344319 0 0) 0x7fa4f8145eb0 con 0x7fa4f81457f0 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e0003520 con 0x7fa4f81457f0 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421317190 0 0) 0x7fa4f80aaef0 con 0x7fa4f813f650 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e00048a0 con 0x7fa4f813f650 2024-03-29T11:04:48.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2574147313 0 0) 0x7fa4e00048a0 con 0x7fa4f813f650 2024-03-29T11:04:48.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 >> v1:172.21.15.107:6790/0 conn(0x7fa4f81457f0 legacy=0x7fa4f813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:48.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.930+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 >> v1:172.21.15.129:6789/0 conn(0x7fa4f80a9d70 legacy=0x7fa4f814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:48.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:04:48.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:04:48.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8146340 con 0x7fa4f813f650 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa4e8001670 con 0x7fa4f813f650 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: dump: 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:48.915 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4fd77b640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4fd77b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4e80017d0 con 0x7fa4f813f650 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 1 -- 172.21.15.107:0/3599819693 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa4e8001be0 con 0x7fa4f813f650 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4f5ffb640 10 monclient: dump: 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:48.916 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4fd77b640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.932+0000 7fa4fd77b640 4 set_mon_vals no callback set 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.933+0000 7fa4fd77b640 10 monclient: shutdown 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.933+0000 7fa4fd77b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.933+0000 7fa4fd77b640 1 -- 172.21.15.107:0/3599819693 >> v1:172.21.15.107:6789/0 conn(0x7fa4f813f650 legacy=0x7fa4f80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.933+0000 7fa4fd77b640 1 -- 172.21.15.107:0/3599819693 shutdown_connections 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.933+0000 7fa4fd77b640 1 -- 172.21.15.107:0/3599819693 >> 172.21.15.107:0/3599819693 conn(0x7fa4f8066a00 msgr2=0x7fa4f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:48.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.933+0000 7fa4fd77b640 1 -- 172.21.15.107:0/3599819693 shutdown_connections 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.933+0000 7fa4fd77b640 1 -- 172.21.15.107:0/3599819693 wait complete. 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.934+0000 7fa4fd77b640 10 monclient: build_initial_monmap 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.934+0000 7fa4fd77b640 10 monclient: monmap: 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:48.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.934+0000 7fa4fd77b640 1 librados: starting msgr at 2024-03-29T11:04:48.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.934+0000 7fa4fd77b640 1 librados: starting objecter 2024-03-29T11:04:48.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.935+0000 7fa4fd77b640 1 Processor -- start 2024-03-29T11:04:48.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.935+0000 7fa4fd77b640 1 -- start start 2024-03-29T11:04:48.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.935+0000 7fa4fd77b640 1 librados: setting wanted keys 2024-03-29T11:04:48.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.935+0000 7fa4fd77b640 1 librados: calling monclient init 2024-03-29T11:04:48.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.935+0000 7fa4fd77b640 10 monclient: init 2024-03-29T11:04:48.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4fd77b640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:48.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4fd77b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4fd77b640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f80a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4f6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa4f80a9d70 0x7fa4f81da730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59096/0 (socket says 172.21.15.107:59096) 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4f6ffd640 1 -- 172.21.15.107:0/1372694667 learned_addr learned my addr 172.21.15.107:0/1372694667 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4fd77b640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f813f650 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4df7fe640 10 client.?.objecter ms_handle_connect 0x7fa4f80a9d70 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4df7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4df7fe640 10 client.?.objecter ms_handle_connect 0x7fa4f81457f0 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4df7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4fd77b640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f81457f0 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f80af1e0 con 0x7fa4f80a9d70 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f80af430 con 0x7fa4f813f650 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.936+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960432537 0 0) 0x7fa4f80af1e0 con 0x7fa4f80a9d70 2024-03-29T11:04:48.926 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f80ad8e0 con 0x7fa4f81457f0 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4fd77b640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4fd77b640 10 monclient(hunting): authenticate will time out at 1220.206579s 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e4004eb0 con 0x7fa4f80a9d70 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903467129 0 0) 0x7fa4f80ad8e0 con 0x7fa4f81457f0 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f80af1e0 con 0x7fa4f81457f0 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.938+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035021648 0 0) 0x7fa4f80af1e0 con 0x7fa4f81457f0 2024-03-29T11:04:48.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4f80ad8e0 con 0x7fa4f81457f0 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa4f0001b30 con 0x7fa4f81457f0 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 10 monclient(hunting): dump: 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:48.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3259400480 0 0) 0x7fa4e4004eb0 con 0x7fa4f80a9d70 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4f80af1e0 con 0x7fa4f80a9d70 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 181161793 0 0) 0x7fa4f80ad8e0 con 0x7fa4f81457f0 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.939+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 >> v1:172.21.15.129:6789/0 conn(0x7fa4f813f650 legacy=0x7fa4f8139cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 >> v1:172.21.15.107:6789/0 conn(0x7fa4f80a9d70 legacy=0x7fa4f81da730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 1 monclient: found mon.c 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f80ada70 con 0x7fa4f81457f0 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 10 monclient: _finish_auth 0 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4fd77b640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4fd77b640 10 monclient: _renew_subs 2024-03-29T11:04:48.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4f0002040 con 0x7fa4f81457f0 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4fd77b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f80ae200 con 0x7fa4f81457f0 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4fd77b640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4fd77b640 10 monclient: _renew_subs 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4fd77b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.940+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f81db020 con 0x7fa4f81457f0 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.941+0000 7fa4fd77b640 1 librados: init done 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.941+0000 7fa4df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.941+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa4f0003bb0 con 0x7fa4f81457f0 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.941+0000 7fa4df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.941+0000 7fa4df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.941+0000 7fa4df7fe640 10 monclient: dump: 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:48.930 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.941+0000 7fa4f4ff9640 4 set_mon_vals no callback set 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.942+0000 7fa4fd77b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.942+0000 7fa4fd77b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.942+0000 7fa4fd77b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.942+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d8005080 con 0x7fa4f81457f0 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.944+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fa4f0019b80 con 0x7fa4f81457f0 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.944+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fa4f0083a50 con 0x7fa4f81457f0 2024-03-29T11:04:48.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.944+0000 7fa4df7fe640 10 client.4547.objecter ms_dispatch 0x7fa4f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:48.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.944+0000 7fa4df7fe640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:48.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.944+0000 7fa4df7fe640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:48.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.944+0000 7fa4df7fe640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:04:48.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.945+0000 7fa4f6ffd640 10 monclient: get_auth_request con 0x7fa4e406dda0 auth_method 0 2024-03-29T11:04:48.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.945+0000 7fa4df7fe640 10 client.4547.objecter ms_handle_connect 0x7fa4e406dda0 2024-03-29T11:04:48.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.951+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4f004f330 con 0x7fa4f81457f0 2024-03-29T11:04:48.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.951+0000 7fa4df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:48.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:48.951+0000 7fa4df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:49.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.260+0000 7fa4fd77b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:49.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.260+0000 7fa4fd77b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:49.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.260+0000 7fa4fd77b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:04:49.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.260+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4d8005330 con 0x7fa4f81457f0 2024-03-29T11:04:49.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.261+0000 7fa4df7fe640 1 -- 172.21.15.107:0/1372694667 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fa4f0051260 con 0x7fa4f81457f0 2024-03-29T11:04:49.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.261+0000 7fa4df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:49.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.261+0000 7fa4df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:49.241 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:04:49.241 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:49.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.265+0000 7fa4fd77b640 10 librados: watch_flush enter 2024-03-29T11:04:49.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.265+0000 7fa4fd77b640 10 librados: watch_flush exit 2024-03-29T11:04:49.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.265+0000 7fa4fd77b640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:04:49.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.265+0000 7fa4fd77b640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:04:49.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.265+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 >> v1:172.21.15.107:6816/882194755 conn(0x7fa4e406dda0 legacy=0x7fa4e4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:49.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.265+0000 7fa4fd77b640 10 monclient: shutdown 2024-03-29T11:04:49.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.265+0000 7fa4fd77b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:49.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.265+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 >> v1:172.21.15.107:6790/0 conn(0x7fa4f81457f0 legacy=0x7fa4f81e2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:49.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.266+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 shutdown_connections 2024-03-29T11:04:49.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.266+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 >> 172.21.15.107:0/1372694667 conn(0x7fa4f8066a00 msgr2=0x7fa4f8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:49.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.266+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 shutdown_connections 2024-03-29T11:04:49.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.266+0000 7fa4fd77b640 1 -- 172.21.15.107:0/1372694667 wait complete. 2024-03-29T11:04:49.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:49.266+0000 7fa4fd77b640 1 librados: shutdown 2024-03-29T11:04:49.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:56.066 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:56.215 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.236+0000 7f7cbab3b640 10 monclient: get_monmap_and_config 2024-03-29T11:04:56.215 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.236+0000 7f7cbab3b640 10 monclient: build_initial_monmap 2024-03-29T11:04:56.215 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.236+0000 7f7cbab3b640 10 monclient: monmap: 2024-03-29T11:04:56.215 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:56.215 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:56.216 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:04:56.237089+0000 2024-03-29T11:04:56.216 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:04:56.237089+0000 2024-03-29T11:04:56.216 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:56.216 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:56.216 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:56.216 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:04:56.216 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:04:56.216 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:56.216 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.237+0000 7f7cbab3b640 1 Processor -- start 2024-03-29T11:04:56.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.237+0000 7f7cbab3b640 1 -- start start 2024-03-29T11:04:56.217 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.237+0000 7f7cbab3b640 10 monclient: init 2024-03-29T11:04:56.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:56.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:04:56.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 10 monclient(hunting): picked mon.noname-b con 0x7f7cb41457f0 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:56.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 10 monclient(hunting): picked mon.noname-c con 0x7f7cb413f650 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:56.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 10 monclient(hunting): picked mon.noname-a con 0x7f7cb40a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:56.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb40aaef0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb4145bd0 con 0x7f7cb413f650 2024-03-29T11:04:56.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb4145eb0 con 0x7f7cb41457f0 2024-03-29T11:04:56.218 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:56.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cbab3b640 10 monclient(hunting): authenticate will time out at 1227.506754s 2024-03-29T11:04:56.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cb88d7640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f7cb41457f0 0x7f7cb413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:58412/0 (socket says 172.21.15.107:58412) 2024-03-29T11:04:56.219 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.238+0000 7f7cb88d7640 1 -- 172.21.15.107:0/2644619863 learned_addr learned my addr 172.21.15.107:0/2644619863 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:56.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.239+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048352363 0 0) 0x7f7cb4145eb0 con 0x7f7cb41457f0 2024-03-29T11:04:56.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.239+0000 7f7cb37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:56.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.239+0000 7f7cb37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:56.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.239+0000 7f7cb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:56.221 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.239+0000 7f7cb37fe640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.239+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c94003520 con 0x7f7cb41457f0 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697063394 0 0) 0x7f7cb40aaef0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb4145eb0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866813054 0 0) 0x7f7c94003520 con 0x7f7cb41457f0 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 >> v1:172.21.15.129:6789/0 conn(0x7f7cb413f650 legacy=0x7f7cb40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 >> v1:172.21.15.107:6789/0 conn(0x7f7cb40a9d70 legacy=0x7f7cb414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 monclient: found mon.noname-b 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:56.222 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb4146340 con 0x7f7cb41457f0 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient: _finish_auth 0 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7cac001690 con 0x7f7cb41457f0 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient: dump: 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:56.223 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cbab3b640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cbab3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cac0017f0 con 0x7f7cb41457f0 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cb37fe640 1 -- 172.21.15.107:0/2644619863 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7cac001c40 con 0x7f7cb41457f0 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cbab3b640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.240+0000 7f7cbab3b640 4 set_mon_vals no callback set 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.241+0000 7f7cbab3b640 10 monclient: shutdown 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.241+0000 7f7cb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.241+0000 7f7cb37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.241+0000 7f7cb37fe640 10 monclient: dump: 2024-03-29T11:04:56.224 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:56.225 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.241+0000 7f7cbab3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:56.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.241+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2644619863 >> v1:172.21.15.107:6790/0 conn(0x7f7cb41457f0 legacy=0x7f7cb413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:56.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.241+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2644619863 shutdown_connections 2024-03-29T11:04:56.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.241+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2644619863 >> 172.21.15.107:0/2644619863 conn(0x7f7cb4066a00 msgr2=0x7f7cb4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:56.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.242+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2644619863 shutdown_connections 2024-03-29T11:04:56.226 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.242+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2644619863 wait complete. 2024-03-29T11:04:56.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.242+0000 7f7cbab3b640 10 monclient: build_initial_monmap 2024-03-29T11:04:56.231 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.242+0000 7f7cbab3b640 10 monclient: monmap: 2024-03-29T11:04:56.231 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:04:56.231 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:56.231 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.243+0000 7f7cbab3b640 1 librados: starting msgr at 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.243+0000 7f7cbab3b640 1 librados: starting objecter 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.243+0000 7f7cbab3b640 1 Processor -- start 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.243+0000 7f7cbab3b640 1 -- start start 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.243+0000 7f7cbab3b640 1 librados: setting wanted keys 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.243+0000 7f7cbab3b640 1 librados: calling monclient init 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.243+0000 7f7cbab3b640 10 monclient: init 2024-03-29T11:04:56.232 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.244+0000 7f7cbab3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.244+0000 7f7cbab3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.244+0000 7f7cbab3b640 10 monclient(hunting): picked mon.noname-a con 0x7f7cb40a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.244+0000 7f7cbab3b640 10 monclient(hunting): picked mon.noname-b con 0x7f7cb413f650 addr v1:172.21.15.129:6789/0 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.244+0000 7f7cbab3b640 10 monclient(hunting): picked mon.noname-c con 0x7f7cb41457f0 addr v1:172.21.15.107:6790/0 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.244+0000 7f7cbab3b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb4144bd0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.244+0000 7f7cbab3b640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb4142d70 con 0x7f7cb413f650 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.245+0000 7f7cbab3b640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb40abe10 con 0x7f7cb41457f0 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.245+0000 7f7cbab3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.245+0000 7f7cb88d7640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7cb40a9d70 0x7f7cb4142660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55230/0 (socket says 172.21.15.107:55230) 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.245+0000 7f7cb88d7640 1 -- 172.21.15.107:0/2955500957 learned_addr learned my addr 172.21.15.107:0/2955500957 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.245+0000 7f7cbab3b640 10 monclient(hunting): authenticate will time out at 1227.513099s 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.245+0000 7f7cb17fa640 10 client.?.objecter ms_handle_connect 0x7f7cb40a9d70 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.245+0000 7f7cb17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.246+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148943385 0 0) 0x7f7cb4144bd0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.233 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.246+0000 7f7cb17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.246+0000 7f7cb17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.246+0000 7f7cb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.246+0000 7f7cb17fa640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.246+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c98004eb0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.246+0000 7f7cb17fa640 10 client.?.objecter ms_handle_connect 0x7f7cb41457f0 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.246+0000 7f7cb17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.247+0000 7f7cb17fa640 10 client.?.objecter ms_handle_connect 0x7f7cb413f650 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.247+0000 7f7cb17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.247+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862302538 0 0) 0x7f7cb40abe10 con 0x7f7cb41457f0 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.247+0000 7f7cb17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.247+0000 7f7cb17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.247+0000 7f7cb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.247+0000 7f7cb17fa640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:04:56.234 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.247+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb4144bd0 con 0x7f7cb41457f0 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904480233 0 0) 0x7f7cb4142d70 con 0x7f7cb413f650 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb40abe10 con 0x7f7cb413f650 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3820017626 0 0) 0x7f7c98004eb0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cb4142d70 con 0x7f7cb40a9d70 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7cac001b40 con 0x7f7cb40a9d70 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): dump: 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.235 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1629122002 0 0) 0x7f7cb40abe10 con 0x7f7cb413f650 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c98004eb0 con 0x7f7cb413f650 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7ca4001760 con 0x7f7cb413f650 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient(hunting): dump: 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:56.236 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 203424444 0 0) 0x7f7cb4142d70 con 0x7f7cb40a9d70 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 >> v1:172.21.15.107:6790/0 conn(0x7f7cb41457f0 legacy=0x7f7cb41e2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 >> v1:172.21.15.129:6789/0 conn(0x7f7cb413f650 legacy=0x7f7cb41370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 monclient: found mon.a 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:56.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb40ac0f0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient: _finish_auth 0 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.248+0000 7f7cb17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.249+0000 7f7cbab3b640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.249+0000 7f7cbab3b640 10 monclient: _renew_subs 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.249+0000 7f7cbab3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.249+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb40ac3d0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.249+0000 7f7cbab3b640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.249+0000 7f7cbab3b640 10 monclient: _renew_subs 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.249+0000 7f7cbab3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.249+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb40abb10 con 0x7f7cb40a9d70 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.249+0000 7f7cbab3b640 1 librados: init done 2024-03-29T11:04:56.238 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.250+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cac0011f0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.250+0000 7f7cb17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.250+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7cac004c30 con 0x7f7cb40a9d70 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.250+0000 7f7cb17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.250+0000 7f7cb17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.250+0000 7f7cb17fa640 10 monclient: dump: 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:04:56.239 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:04:56.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.250+0000 7f7cb2ffd640 4 set_mon_vals no callback set 2024-03-29T11:04:56.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.250+0000 7f7cbab3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:56.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.251+0000 7f7cbab3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:56.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.251+0000 7f7cbab3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:56.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.251+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8c005080 con 0x7f7cb40a9d70 2024-03-29T11:04:56.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.252+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f7cac01ac00 con 0x7f7cb40a9d70 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.252+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f7cac084ad0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.252+0000 7f7cb17fa640 10 client.4488.objecter ms_dispatch 0x7f7cb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.252+0000 7f7cb17fa640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.252+0000 7f7cb17fa640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.253+0000 7f7cb17fa640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.253+0000 7f7cb3fff640 10 monclient: get_auth_request con 0x7f7c9806df00 auth_method 0 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.253+0000 7f7cb17fa640 10 client.4488.objecter ms_handle_connect 0x7f7c9806df00 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.260+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cac086e60 con 0x7f7cb40a9d70 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.260+0000 7f7cb17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:56.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.260+0000 7f7cb17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:56.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.614+0000 7f7cbab3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:56.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.614+0000 7f7cbab3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:56.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.614+0000 7f7cbab3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:04:56.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.614+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c8c005330 con 0x7f7cb40a9d70 2024-03-29T11:04:56.595 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:04:56.595 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:56.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.615+0000 7f7cb17fa640 1 -- 172.21.15.107:0/2955500957 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f7cac0501e0 con 0x7f7cb40a9d70 2024-03-29T11:04:56.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.615+0000 7f7cb17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:56.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.615+0000 7f7cb17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:56.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.619+0000 7f7cbab3b640 10 librados: watch_flush enter 2024-03-29T11:04:56.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 10 librados: watch_flush exit 2024-03-29T11:04:56.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T11:04:56.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 10 client.4488.objecter successfully canceled tick 2024-03-29T11:04:56.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 >> v1:172.21.15.107:6816/882194755 conn(0x7f7c9806df00 legacy=0x7f7c980703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:56.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 10 monclient: shutdown 2024-03-29T11:04:56.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:56.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 >> v1:172.21.15.107:6789/0 conn(0x7f7cb40a9d70 legacy=0x7f7cb4142660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:56.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 shutdown_connections 2024-03-29T11:04:56.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 >> 172.21.15.107:0/2955500957 conn(0x7f7cb4066a00 msgr2=0x7f7cb413aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:56.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 shutdown_connections 2024-03-29T11:04:56.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 1 -- 172.21.15.107:0/2955500957 wait complete. 2024-03-29T11:04:56.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:04:56.620+0000 7f7cbab3b640 1 librados: shutdown 2024-03-29T11:04:56.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:03.423 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:03.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.593+0000 7fc9bacf6640 10 monclient: get_monmap_and_config 2024-03-29T11:05:03.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.593+0000 7fc9bacf6640 10 monclient: build_initial_monmap 2024-03-29T11:05:03.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.593+0000 7fc9bacf6640 10 monclient: monmap: 2024-03-29T11:05:03.573 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:03.573 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:03.573 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:05:03.594515+0000 2024-03-29T11:05:03.573 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:05:03.594515+0000 2024-03-29T11:05:03.574 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:03.574 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:03.574 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:03.574 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:05:03.574 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:05:03.574 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:03.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.595+0000 7fc9bacf6640 1 Processor -- start 2024-03-29T11:05:03.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.595+0000 7fc9bacf6640 1 -- start start 2024-03-29T11:05:03.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.595+0000 7fc9bacf6640 10 monclient: init 2024-03-29T11:05:03.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:03.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:05:03.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 10 monclient(hunting): picked mon.noname-c con 0x7fc9b40a9930 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:03.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9b4149dc0 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:03.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 10 monclient(hunting): picked mon.noname-b con 0x7fc9b414ff40 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:03.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b40aae60 con 0x7fc9b4149dc0 2024-03-29T11:05:03.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b40b4310 con 0x7fc9b40a9930 2024-03-29T11:05:03.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b40afdd0 con 0x7fc9b414ff40 2024-03-29T11:05:03.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:03.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.596+0000 7fc9bacf6640 10 monclient(hunting): authenticate will time out at 1234.864348s 2024-03-29T11:05:03.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.599+0000 7fc9b8a92640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fc9b40a9930 0x7fc9b4143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:32835/0 (socket says 172.21.15.107:32835) 2024-03-29T11:05:03.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.599+0000 7fc9b8a92640 1 -- 172.21.15.107:0/2392285100 learned_addr learned my addr 172.21.15.107:0/2392285100 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:03.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124114937 0 0) 0x7fc9b40b4310 con 0x7fc9b40a9930 2024-03-29T11:05:03.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:05:03.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:03.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:03.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:05:03.583 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc99c003520 con 0x7fc9b40a9930 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491567410 0 0) 0x7fc9b40afdd0 con 0x7fc9b414ff40 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.600+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b40b4310 con 0x7fc9b414ff40 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1643958391 0 0) 0x7fc9b40b4310 con 0x7fc9b414ff40 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 >> v1:172.21.15.129:6789/0 conn(0x7fc9b40a9930 legacy=0x7fc9b4143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 >> v1:172.21.15.107:6789/0 conn(0x7fc9b4149dc0 legacy=0x7fc9b414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 1 monclient: found mon.noname-b 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:05:03.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b414a950 con 0x7fc9b414ff40 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: _finish_auth 0 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc9ac001970 con 0x7fc9b414ff40 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: dump: 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:03.585 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9ac001b50 con 0x7fc9b414ff40 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 1 -- 172.21.15.107:0/2392285100 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc9ac001fa0 con 0x7fc9b414ff40 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9b37fe640 10 monclient: dump: 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:03.586 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9bacf6640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9bacf6640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.601+0000 7fc9bacf6640 4 set_mon_vals no callback set 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.602+0000 7fc9bacf6640 10 monclient: shutdown 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.602+0000 7fc9bacf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.602+0000 7fc9bacf6640 1 -- 172.21.15.107:0/2392285100 >> v1:172.21.15.107:6790/0 conn(0x7fc9b414ff40 legacy=0x7fc9b4152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.602+0000 7fc9bacf6640 1 -- 172.21.15.107:0/2392285100 shutdown_connections 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.602+0000 7fc9bacf6640 1 -- 172.21.15.107:0/2392285100 >> 172.21.15.107:0/2392285100 conn(0x7fc9b40b35e0 msgr2=0x7fc9b40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.603+0000 7fc9bacf6640 1 -- 172.21.15.107:0/2392285100 shutdown_connections 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.603+0000 7fc9bacf6640 1 -- 172.21.15.107:0/2392285100 wait complete. 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.604+0000 7fc9bacf6640 10 monclient: build_initial_monmap 2024-03-29T11:05:03.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.604+0000 7fc9bacf6640 10 monclient: monmap: 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:05:03.588 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.604+0000 7fc9bacf6640 1 librados: starting msgr at 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.604+0000 7fc9bacf6640 1 librados: starting objecter 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.604+0000 7fc9bacf6640 1 Processor -- start 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.604+0000 7fc9bacf6640 1 -- start start 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.604+0000 7fc9bacf6640 1 librados: setting wanted keys 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.604+0000 7fc9bacf6640 1 librados: calling monclient init 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.604+0000 7fc9bacf6640 10 monclient: init 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.605+0000 7fc9bacf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.605+0000 7fc9bacf6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.606+0000 7fc9bacf6640 10 monclient(hunting): picked mon.noname-b con 0x7fc9b40a9930 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.606+0000 7fc9bacf6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9b4149dc0 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:03.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.606+0000 7fc9bacf6640 10 monclient(hunting): picked mon.noname-c con 0x7fc9b414ff40 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.606+0000 7fc9bacf6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b4144640 con 0x7fc9b4149dc0 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.606+0000 7fc9bacf6640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b4148ee0 con 0x7fc9b40a9930 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.606+0000 7fc9bacf6640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b4144bf0 con 0x7fc9b414ff40 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.606+0000 7fc9bacf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.606+0000 7fc9bacf6640 10 monclient(hunting): authenticate will time out at 1234.874343s 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.608+0000 7fc9b9293640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc9b414ff40 0x7fc9b41e6e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40584/0 (socket says 172.21.15.107:40584) 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.608+0000 7fc9b9293640 1 -- 172.21.15.107:0/454888722 learned_addr learned my addr 172.21.15.107:0/454888722 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 client.?.objecter ms_handle_connect 0x7fc9b414ff40 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642846612 0 0) 0x7fc9b4144bf0 con 0x7fc9b414ff40 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:03.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a0004eb0 con 0x7fc9b414ff40 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908381982 0 0) 0x7fc9a0004eb0 con 0x7fc9b414ff40 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9b4144bf0 con 0x7fc9b414ff40 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc9ac001fa0 con 0x7fc9b414ff40 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 monclient(hunting): dump: 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:03.597 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.609+0000 7fc9b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2822112012 0 0) 0x7fc9b4144bf0 con 0x7fc9b414ff40 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 >> v1:172.21.15.129:6789/0 conn(0x7fc9b40a9930 legacy=0x7fc9b413b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 >> v1:172.21.15.107:6789/0 conn(0x7fc9b4149dc0 legacy=0x7fc9b41dead0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 1 monclient: found mon.c 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4144d80 con 0x7fc9b414ff40 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 10 monclient: _finish_auth 0 2024-03-29T11:05:03.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9bacf6640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9bacf6640 10 monclient: _renew_subs 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9bacf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b4149170 con 0x7fc9b414ff40 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9bacf6640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9bacf6640 10 monclient: _renew_subs 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9bacf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b413bc30 con 0x7fc9b414ff40 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.610+0000 7fc9bacf6640 1 librados: init done 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.611+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9ac003c20 con 0x7fc9b414ff40 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.611+0000 7fc9b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.611+0000 7fc9b2ffd640 4 set_mon_vals no callback set 2024-03-29T11:05:03.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.612+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc9ac004030 con 0x7fc9b414ff40 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.612+0000 7fc9b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.612+0000 7fc9b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.612+0000 7fc9b17fa640 10 monclient: dump: 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.613+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fc9ac004210 con 0x7fc9b414ff40 2024-03-29T11:05:03.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.613+0000 7fc9bacf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.613+0000 7fc9bacf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.613+0000 7fc9bacf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.613+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc994005080 con 0x7fc9b414ff40 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.614+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fc9ac085060 con 0x7fc9b414ff40 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.614+0000 7fc9b17fa640 10 client.4559.objecter ms_dispatch 0x7fc9b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.614+0000 7fc9b17fa640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.614+0000 7fc9b17fa640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.614+0000 7fc9b17fa640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.614+0000 7fc9b8a92640 10 monclient: get_auth_request con 0x7fc9a006dba0 auth_method 0 2024-03-29T11:05:03.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.614+0000 7fc9b17fa640 10 client.4559.objecter ms_handle_connect 0x7fc9a006dba0 2024-03-29T11:05:03.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.622+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9ac050940 con 0x7fc9b414ff40 2024-03-29T11:05:03.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.622+0000 7fc9b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:03.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.622+0000 7fc9b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:03.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.936+0000 7fc9bacf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:03.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.936+0000 7fc9bacf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:03.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.936+0000 7fc9bacf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:03.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.936+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc994005330 con 0x7fc9b414ff40 2024-03-29T11:05:03.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.937+0000 7fc9b17fa640 1 -- 172.21.15.107:0/454888722 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fc9ac052870 con 0x7fc9b414ff40 2024-03-29T11:05:03.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.937+0000 7fc9b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:03.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.937+0000 7fc9b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:03.916 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:05:03.917 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:03.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.940+0000 7fc9bacf6640 10 librados: watch_flush enter 2024-03-29T11:05:03.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.940+0000 7fc9bacf6640 10 librados: watch_flush exit 2024-03-29T11:05:03.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.940+0000 7fc9bacf6640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:05:03.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.940+0000 7fc9bacf6640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:05:03.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.941+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 >> v1:172.21.15.107:6816/882194755 conn(0x7fc9a006dba0 legacy=0x7fc9a0070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:03.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.941+0000 7fc9bacf6640 10 monclient: shutdown 2024-03-29T11:05:03.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.941+0000 7fc9bacf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:03.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.941+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 >> v1:172.21.15.107:6790/0 conn(0x7fc9b414ff40 legacy=0x7fc9b41e6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:03.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.941+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 shutdown_connections 2024-03-29T11:05:03.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.941+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 >> 172.21.15.107:0/454888722 conn(0x7fc9b40b35e0 msgr2=0x7fc9b4140950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:03.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.941+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 shutdown_connections 2024-03-29T11:05:03.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.942+0000 7fc9bacf6640 1 -- 172.21.15.107:0/454888722 wait complete. 2024-03-29T11:05:03.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:03.942+0000 7fc9bacf6640 1 librados: shutdown 2024-03-29T11:05:03.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:10.742 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:10.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.909+0000 7f554a61e640 10 monclient: get_monmap_and_config 2024-03-29T11:05:10.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.909+0000 7f554a61e640 10 monclient: build_initial_monmap 2024-03-29T11:05:10.889 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.910+0000 7f554a61e640 10 monclient: monmap: 2024-03-29T11:05:10.889 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:10.889 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:10.889 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:05:10.910958+0000 2024-03-29T11:05:10.889 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:05:10.910958+0000 2024-03-29T11:05:10.890 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:10.890 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:10.890 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:10.890 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:05:10.890 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:05:10.890 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:10.892 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.912+0000 7f554a61e640 1 Processor -- start 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.912+0000 7f554a61e640 1 -- start start 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.912+0000 7f554a61e640 10 monclient: init 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.913+0000 7f554a61e640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.913+0000 7f554a61e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.913+0000 7f554a61e640 10 monclient(hunting): picked mon.noname-b con 0x7f55440b1f60 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.913+0000 7f554a61e640 10 monclient(hunting): picked mon.noname-a con 0x7f5544071eb0 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.913+0000 7f554a61e640 10 monclient(hunting): picked mon.noname-c con 0x7f5544143d10 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.913+0000 7f554a61e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554406ef80 con 0x7f5544071eb0 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.913+0000 7f554a61e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554406f260 con 0x7f5544143d10 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.913+0000 7f554a61e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55440b40c0 con 0x7f55440b1f60 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.913+0000 7f554a61e640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:10.893 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.914+0000 7f554a61e640 10 monclient(hunting): authenticate will time out at 1242.181988s 2024-03-29T11:05:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.916+0000 7f5548bbb640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f5544143d10 0x7f5544146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:59435/0 (socket says 172.21.15.107:59435) 2024-03-29T11:05:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.916+0000 7f5548bbb640 1 -- 172.21.15.107:0/3754385005 learned_addr learned my addr 172.21.15.107:0/3754385005 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.917+0000 7f5542ffd640 1 -- 172.21.15.107:0/3754385005 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352554957 0 0) 0x7f554406f260 con 0x7f5544143d10 2024-03-29T11:05:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.917+0000 7f5542ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:05:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.917+0000 7f5542ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.917+0000 7f5542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:10.896 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.917+0000 7f5542ffd640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:05:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.917+0000 7f5542ffd640 1 -- 172.21.15.107:0/3754385005 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f552c003520 con 0x7f5544143d10 2024-03-29T11:05:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.917+0000 7f5542ffd640 1 -- 172.21.15.107:0/3754385005 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3867093763 0 0) 0x7f552c003520 con 0x7f5544143d10 2024-03-29T11:05:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.917+0000 7f5542ffd640 1 -- 172.21.15.107:0/3754385005 >> v1:172.21.15.107:6790/0 conn(0x7f55440b1f60 legacy=0x7f5544143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:05:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.917+0000 7f5542ffd640 1 -- 172.21.15.107:0/3754385005 >> v1:172.21.15.107:6789/0 conn(0x7f5544071eb0 legacy=0x7f5544146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:05:10.897 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 1 monclient: found mon.noname-c 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 1 -- 172.21.15.107:0/3754385005 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5544155d70 con 0x7f5544143d10 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: _finish_auth 0 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f554a61e640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f554a61e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 1 -- 172.21.15.107:0/3754385005 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f553c001a10 con 0x7f5544143d10 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:10.898 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: dump: 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f554a61e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 1 -- 172.21.15.107:0/3754385005 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f553c001ca0 con 0x7f5544143d10 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f554a61e640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:10.899 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f554a61e640 4 set_mon_vals no callback set 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 1 -- 172.21.15.107:0/3754385005 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f553c002030 con 0x7f5544143d10 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.918+0000 7f5542ffd640 10 monclient: dump: 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:10.900 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:10.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.919+0000 7f554a61e640 10 monclient: shutdown 2024-03-29T11:05:10.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.919+0000 7f554a61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:10.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.919+0000 7f554a61e640 1 -- 172.21.15.107:0/3754385005 >> v1:172.21.15.129:6789/0 conn(0x7f5544143d10 legacy=0x7f5544146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:10.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.919+0000 7f554a61e640 1 -- 172.21.15.107:0/3754385005 shutdown_connections 2024-03-29T11:05:10.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.919+0000 7f554a61e640 1 -- 172.21.15.107:0/3754385005 >> 172.21.15.107:0/3754385005 conn(0x7f5544066a00 msgr2=0x7f5544066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:10.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.919+0000 7f554a61e640 1 -- 172.21.15.107:0/3754385005 shutdown_connections 2024-03-29T11:05:10.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.919+0000 7f554a61e640 1 -- 172.21.15.107:0/3754385005 wait complete. 2024-03-29T11:05:10.902 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.920+0000 7f554a61e640 10 monclient: build_initial_monmap 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.920+0000 7f554a61e640 10 monclient: monmap: 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.920+0000 7f554a61e640 1 librados: starting msgr at 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.920+0000 7f554a61e640 1 librados: starting objecter 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.921+0000 7f554a61e640 1 Processor -- start 2024-03-29T11:05:10.903 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.921+0000 7f554a61e640 1 -- start start 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.921+0000 7f554a61e640 1 librados: setting wanted keys 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.921+0000 7f554a61e640 1 librados: calling monclient init 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.921+0000 7f554a61e640 10 monclient: init 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 10 monclient(hunting): picked mon.noname-a con 0x7f55440b1f60 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 10 monclient(hunting): picked mon.noname-c con 0x7f5544143d10 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 10 monclient(hunting): picked mon.noname-b con 0x7f55441e4c70 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55440aca40 con 0x7f55440b1f60 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55440accb0 con 0x7f55441e4c70 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55441e82a0 con 0x7f5544143d10 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:10.904 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.922+0000 7f554a61e640 10 monclient(hunting): authenticate will time out at 1242.190808s 2024-03-29T11:05:10.906 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.923+0000 7f55437fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f5544143d10 0x7f55441e4560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:46358/0 (socket says 172.21.15.107:46358) 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.923+0000 7f55437fe640 1 -- 172.21.15.107:0/3081818394 learned_addr learned my addr 172.21.15.107:0/3081818394 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 10 client.?.objecter ms_handle_connect 0x7f5544143d10 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 10 client.?.objecter ms_handle_connect 0x7f55440b1f60 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117155034 0 0) 0x7f55441e82a0 con 0x7f5544143d10 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5530004eb0 con 0x7f5544143d10 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.925+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904905922 0 0) 0x7f55440aca40 con 0x7f55440b1f60 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.926+0000 7f5540ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.926+0000 7f5540ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:10.907 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.926+0000 7f5540ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.926+0000 7f5540ff9640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.926+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55441e82a0 con 0x7f55440b1f60 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.926+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4026040338 0 0) 0x7f5530004eb0 con 0x7f5544143d10 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.926+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55440aca40 con 0x7f5544143d10 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.927+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f5538004080 con 0x7f5544143d10 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.927+0000 7f5540ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.927+0000 7f5540ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.927+0000 7f5540ff9640 10 monclient(hunting): dump: 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:10.908 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:10.909 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:10.909 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:10.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.927+0000 7f5540ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:10.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.927+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3186942485 0 0) 0x7f55440aca40 con 0x7f5544143d10 2024-03-29T11:05:10.909 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.927+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 >> v1:172.21.15.129:6789/0 conn(0x7f55441e4c70 legacy=0x7f55441eefa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:10.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.929+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 >> v1:172.21.15.107:6789/0 conn(0x7f55440b1f60 legacy=0x7f55441e7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:10.911 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.929+0000 7f5540ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.929+0000 7f5540ff9640 1 monclient: found mon.c 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.929+0000 7f5540ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.929+0000 7f5540ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.929+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55441e87c0 con 0x7f5544143d10 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.929+0000 7f5540ff9640 10 monclient: _finish_auth 0 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.929+0000 7f5540ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.929+0000 7f5540ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55380041e0 con 0x7f5544143d10 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f554a61e640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f554a61e640 10 monclient: _renew_subs 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f554a61e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55440708e0 con 0x7f5544143d10 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f5540ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:10.912 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f5538001180 con 0x7f5544143d10 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f5540ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f5540ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f5540ff9640 10 monclient: dump: 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f55427fc640 4 set_mon_vals no callback set 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f554a61e640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:10.913 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f554a61e640 10 monclient: _renew_subs 2024-03-29T11:05:10.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f554a61e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:10.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5544070c80 con 0x7f5544143d10 2024-03-29T11:05:10.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.930+0000 7f554a61e640 1 librados: init done 2024-03-29T11:05:10.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.932+0000 7f554a61e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:10.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.932+0000 7f554a61e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:10.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.932+0000 7f554a61e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:10.914 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.932+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5544075f10 con 0x7f5544143d10 2024-03-29T11:05:10.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.932+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f55380012e0 con 0x7f5544143d10 2024-03-29T11:05:10.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.933+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f5538084a70 con 0x7f5544143d10 2024-03-29T11:05:10.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.934+0000 7f5540ff9640 10 client.4562.objecter ms_dispatch 0x7f5544000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:10.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.934+0000 7f5540ff9640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:10.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.934+0000 7f5540ff9640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:10.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.934+0000 7f5540ff9640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:05:10.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.934+0000 7f5543fff640 10 monclient: get_auth_request con 0x7f553006db40 auth_method 0 2024-03-29T11:05:10.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.935+0000 7f5540ff9640 10 client.4562.objecter ms_handle_connect 0x7f553006db40 2024-03-29T11:05:10.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.942+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5538050180 con 0x7f5544143d10 2024-03-29T11:05:10.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.942+0000 7f5540ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:10.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:10.942+0000 7f5540ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:11.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.283+0000 7f554a61e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:11.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.283+0000 7f554a61e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:11.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.283+0000 7f554a61e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:11.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.283+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5544070a70 con 0x7f5544143d10 2024-03-29T11:05:11.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.284+0000 7f5540ff9640 1 -- 172.21.15.107:0/3081818394 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f5538052280 con 0x7f5544143d10 2024-03-29T11:05:11.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.284+0000 7f5540ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:11.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.284+0000 7f5540ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:11.264 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:05:11.264 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:11.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.288+0000 7f554a61e640 10 librados: watch_flush enter 2024-03-29T11:05:11.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.288+0000 7f554a61e640 10 librados: watch_flush exit 2024-03-29T11:05:11.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.288+0000 7f554a61e640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:05:11.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.288+0000 7f554a61e640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:05:11.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.288+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 >> v1:172.21.15.107:6816/882194755 conn(0x7f553006db40 legacy=0x7f5530070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:11.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.288+0000 7f554a61e640 10 monclient: shutdown 2024-03-29T11:05:11.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.288+0000 7f554a61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.288+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 >> v1:172.21.15.107:6790/0 conn(0x7f5544143d10 legacy=0x7f55441e4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.289+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 shutdown_connections 2024-03-29T11:05:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.289+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 >> 172.21.15.107:0/3081818394 conn(0x7f5544066a00 msgr2=0x7f55440b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.290+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 shutdown_connections 2024-03-29T11:05:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.290+0000 7f554a61e640 1 -- 172.21.15.107:0/3081818394 wait complete. 2024-03-29T11:05:11.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:11.290+0000 7f554a61e640 1 librados: shutdown 2024-03-29T11:05:11.287 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:18.094 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:18.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.261+0000 7f9dcfb82640 10 monclient: get_monmap_and_config 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.261+0000 7f9dcfb82640 10 monclient: build_initial_monmap 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.261+0000 7f9dcfb82640 10 monclient: monmap: 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:05:18.262189+0000 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:05:18.262189+0000 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:05:18.241 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:05:18.242 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.262+0000 7f9dcfb82640 1 Processor -- start 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.262+0000 7f9dcfb82640 1 -- start start 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.262+0000 7f9dcfb82640 10 monclient: init 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 10 monclient(hunting): picked mon.noname-b con 0x7f9dc81422b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc8142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc80b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc8142c80 con 0x7f9dc8142690 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc80aae40 con 0x7f9dc80b17d0 2024-03-29T11:05:18.243 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc813fbd0 con 0x7f9dc81422b0 2024-03-29T11:05:18.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:18.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.263+0000 7f9dcfb82640 10 monclient(hunting): authenticate will time out at 1249.531347s 2024-03-29T11:05:18.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.266+0000 7f9dce11f640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f9dc80b17d0 0x7f9dc80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:53447/0 (socket says 172.21.15.107:53447) 2024-03-29T11:05:18.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.266+0000 7f9dce11f640 1 -- 172.21.15.107:0/3547706887 learned_addr learned my addr 172.21.15.107:0/3547706887 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215493094 0 0) 0x7f9dc80aae40 con 0x7f9dc80b17d0 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 10 monclient(hunting): _negotiate 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9db4003520 con 0x7f9dc80b17d0 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459492861 0 0) 0x7f9dc813fbd0 con 0x7f9dc81422b0 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 10 monclient(hunting): _negotiate 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:18.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:05:18.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc80aae40 con 0x7f9dc81422b0 2024-03-29T11:05:18.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4207663837 0 0) 0x7f9db4003520 con 0x7f9dc80b17d0 2024-03-29T11:05:18.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 >> v1:172.21.15.107:6790/0 conn(0x7f9dc81422b0 legacy=0x7f9dc813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:18.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.267+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 >> v1:172.21.15.107:6789/0 conn(0x7f9dc8142690 legacy=0x7f9dc80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:05:18.250 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 10 monclient: _finish_hunting 0 2024-03-29T11:05:18.250 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 1 monclient: found mon.noname-c 2024-03-29T11:05:18.250 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc8151940 con 0x7f9dc80b17d0 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 10 monclient: _finish_auth 0 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 10 monclient: _check_auth_tickets 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcfb82640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcfb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9dc0001a10 con 0x7f9dc80b17d0 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 10 monclient: dump: 2024-03-29T11:05:18.251 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcfb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dc0001c30 con 0x7f9dc80b17d0 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcc91c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcfb82640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:18.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.271+0000 7f9dcfb82640 4 set_mon_vals no callback set 2024-03-29T11:05:18.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.272+0000 7f9dcc91c640 1 -- 172.21.15.107:0/3547706887 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9dc0001f40 con 0x7f9dc80b17d0 2024-03-29T11:05:18.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.272+0000 7f9dcc91c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:18.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.272+0000 7f9dcc91c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:18.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.272+0000 7f9dcc91c640 10 monclient: dump: 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.272+0000 7f9dcfb82640 10 monclient: shutdown 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.272+0000 7f9dcfb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.272+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3547706887 >> v1:172.21.15.129:6789/0 conn(0x7f9dc80b17d0 legacy=0x7f9dc80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:18.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.272+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3547706887 shutdown_connections 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.272+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3547706887 >> 172.21.15.107:0/3547706887 conn(0x7f9dc8066a00 msgr2=0x7f9dc8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.273+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3547706887 shutdown_connections 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.273+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3547706887 wait complete. 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.274+0000 7f9dcfb82640 10 monclient: build_initial_monmap 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.274+0000 7f9dcfb82640 10 monclient: monmap: 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:05:18.256 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:05:18.257 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:18.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.275+0000 7f9dcfb82640 1 librados: starting msgr at 2024-03-29T11:05:18.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.275+0000 7f9dcfb82640 1 librados: starting objecter 2024-03-29T11:05:18.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.275+0000 7f9dcfb82640 1 Processor -- start 2024-03-29T11:05:18.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.275+0000 7f9dcfb82640 1 -- start start 2024-03-29T11:05:18.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.275+0000 7f9dcfb82640 1 librados: setting wanted keys 2024-03-29T11:05:18.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.275+0000 7f9dcfb82640 1 librados: calling monclient init 2024-03-29T11:05:18.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.275+0000 7f9dcfb82640 10 monclient: init 2024-03-29T11:05:18.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:18.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:05:18.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc80b17d0 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:18.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc81422b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:18.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 10 monclient(hunting): picked mon.noname-b con 0x7f9dc8142690 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc80abb70 con 0x7f9dc80b17d0 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc80ac930 con 0x7f9dc8142690 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc80acc10 con 0x7f9dc81422b0 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.276+0000 7f9dcfb82640 10 monclient(hunting): authenticate will time out at 1249.544098s 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dcd11d640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f9dc81422b0 0x7f9dc8130690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:51024/0 (socket says 172.21.15.107:51024) 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dcd11d640 1 -- 172.21.15.107:0/3888751445 learned_addr learned my addr 172.21.15.107:0/3888751445 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 10 client.?.objecter ms_handle_connect 0x7f9dc81422b0 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 10 client.?.objecter ms_handle_connect 0x7f9dc80b17d0 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510379115 0 0) 0x7f9dc80acc10 con 0x7f9dc81422b0 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:18.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dac004eb0 con 0x7f9dc81422b0 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717446634 0 0) 0x7f9dc80abb70 con 0x7f9dc80b17d0 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.278+0000 7f9dbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc80acc10 con 0x7f9dc80b17d0 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3407627216 0 0) 0x7f9dc80acc10 con 0x7f9dc80b17d0 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dc80abb70 con 0x7f9dc80b17d0 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9dc4001930 con 0x7f9dc80b17d0 2024-03-29T11:05:18.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 10 monclient(hunting): dump: 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:18.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 365492357 0 0) 0x7f9dc80abb70 con 0x7f9dc80b17d0 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 >> v1:172.21.15.107:6790/0 conn(0x7f9dc81422b0 legacy=0x7f9dc8130690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.279+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 >> v1:172.21.15.129:6789/0 conn(0x7f9dc8142690 legacy=0x7f9dc81e1d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 1 monclient: found mon.a 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc80abe50 con 0x7f9dc80b17d0 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dcfb82640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dcfb82640 10 monclient: _renew_subs 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dcfb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:18.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc8132910 con 0x7f9dc80b17d0 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dcfb82640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dcfb82640 10 monclient: _renew_subs 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dcfb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc8132bc0 con 0x7f9dc80b17d0 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dcfb82640 1 librados: init done 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dc4003610 con 0x7f9dc80b17d0 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9dc4003920 con 0x7f9dc80b17d0 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.282+0000 7f9dbe7fc640 10 monclient: dump: 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:18.268 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:18.269 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:18.269 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:18.269 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:18.269 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:18.269 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:18.269 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:18.269 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:18.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.283+0000 7f9dbf7fe640 4 set_mon_vals no callback set 2024-03-29T11:05:18.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.283+0000 7f9dcfb82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:18.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.284+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f9dc40198f0 con 0x7f9dc80b17d0 2024-03-29T11:05:18.270 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.285+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f9dc40837f0 con 0x7f9dc80b17d0 2024-03-29T11:05:18.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.285+0000 7f9dbe7fc640 10 client.4494.objecter ms_dispatch 0x7f9dc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:18.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.285+0000 7f9dbe7fc640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:18.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.285+0000 7f9dbe7fc640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:18.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.285+0000 7f9dbe7fc640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-29T11:05:18.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.287+0000 7f9dcfb82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:18.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.287+0000 7f9dcfb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:18.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.287+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db0005080 con 0x7f9dc80b17d0 2024-03-29T11:05:18.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.287+0000 7f9dcd11d640 10 monclient: get_auth_request con 0x7f9dac06dd70 auth_method 0 2024-03-29T11:05:18.271 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.287+0000 7f9dbe7fc640 10 client.4494.objecter ms_handle_connect 0x7f9dac06dd70 2024-03-29T11:05:18.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.295+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9dc404ef00 con 0x7f9dc80b17d0 2024-03-29T11:05:18.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.295+0000 7f9dbe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:18.274 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.295+0000 7f9dbe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:18.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.617+0000 7f9dcfb82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:18.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.617+0000 7f9dcfb82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:18.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.617+0000 7f9dcfb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:18.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.617+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9db0005330 con 0x7f9dc80b17d0 2024-03-29T11:05:18.598 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.619+0000 7f9dbe7fc640 1 -- 172.21.15.107:0/3888751445 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f9dc4051000 con 0x7f9dc80b17d0 2024-03-29T11:05:18.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.619+0000 7f9dbe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:18.599 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.619+0000 7f9dbe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:18.599 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:05:18.599 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:18.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.623+0000 7f9dcfb82640 10 librados: watch_flush enter 2024-03-29T11:05:18.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.625+0000 7f9dcfb82640 10 librados: watch_flush exit 2024-03-29T11:05:18.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.625+0000 7f9dcfb82640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-29T11:05:18.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.625+0000 7f9dcfb82640 10 client.4494.objecter successfully canceled tick 2024-03-29T11:05:18.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.625+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 >> v1:172.21.15.107:6816/882194755 conn(0x7f9dac06dd70 legacy=0x7f9dac070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:18.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.625+0000 7f9dcfb82640 10 monclient: shutdown 2024-03-29T11:05:18.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.626+0000 7f9dcfb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:18.605 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.626+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 >> v1:172.21.15.107:6789/0 conn(0x7f9dc80b17d0 legacy=0x7f9dc812ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:18.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.628+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 shutdown_connections 2024-03-29T11:05:18.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.629+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 >> 172.21.15.107:0/3888751445 conn(0x7f9dc8066a00 msgr2=0x7f9dc80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:18.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.629+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 shutdown_connections 2024-03-29T11:05:18.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.629+0000 7f9dcfb82640 1 -- 172.21.15.107:0/3888751445 wait complete. 2024-03-29T11:05:18.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:18.630+0000 7f9dcfb82640 1 librados: shutdown 2024-03-29T11:05:18.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:25.432 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.601+0000 7f75d3b6b640 10 monclient: get_monmap_and_config 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.601+0000 7f75d3b6b640 10 monclient: build_initial_monmap 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.601+0000 7f75d3b6b640 10 monclient: monmap: 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:05:25.602541+0000 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:05:25.602541+0000 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:05:25.581 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:05:25.582 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:25.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.602+0000 7f75d3b6b640 1 Processor -- start 2024-03-29T11:05:25.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.602+0000 7f75d3b6b640 1 -- start start 2024-03-29T11:05:25.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.602+0000 7f75d3b6b640 10 monclient: init 2024-03-29T11:05:25.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.603+0000 7f75d3b6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:25.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.603+0000 7f75d3b6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:25.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.603+0000 7f75d3b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f75cc0b0770 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:25.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.603+0000 7f75d3b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f75cc14a700 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:25.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.606+0000 7f75d1907640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f75cc0b0770 0x7f75cc149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:53592/0 (socket says 172.21.15.107:53592) 2024-03-29T11:05:25.586 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.606+0000 7f75d1907640 1 -- 172.21.15.107:0/420707262 learned_addr learned my addr 172.21.15.107:0/420707262 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:25.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.610+0000 7f75d3b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f75cc0ac5a0 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:25.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.610+0000 7f75d3b6b640 1 -- 172.21.15.107:0/420707262 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0aba70 con 0x7f75cc14a700 2024-03-29T11:05:25.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.610+0000 7f75d3b6b640 1 -- 172.21.15.107:0/420707262 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0aecb0 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.610+0000 7f75d3b6b640 1 -- 172.21.15.107:0/420707262 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0af150 con 0x7f75cc0b0770 2024-03-29T11:05:25.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d3b6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:25.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d3b6b640 10 monclient(hunting): authenticate will time out at 1256.879097s 2024-03-29T11:05:25.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588951095 0 0) 0x7f75cc0aecb0 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d0905640 10 monclient(hunting): _negotiate 2024-03-29T11:05:25.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d0905640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:25.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d0905640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:25.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d0905640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:05:25.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8003520 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331302808 0 0) 0x7f75cc0af150 con 0x7f75cc0b0770 2024-03-29T11:05:25.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d0905640 10 monclient(hunting): _negotiate 2024-03-29T11:05:25.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.611+0000 7f75d0905640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:25.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75cc0aecb0 con 0x7f75cc0b0770 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574576646 0 0) 0x7f75cc0aba70 con 0x7f75cc14a700 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient(hunting): _negotiate 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75cc0af150 con 0x7f75cc14a700 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045179255 0 0) 0x7f75cc0aecb0 con 0x7f75cc0b0770 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 >> v1:172.21.15.129:6789/0 conn(0x7f75cc0ac5a0 legacy=0x7f75cc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 >> v1:172.21.15.107:6789/0 conn(0x7f75cc14a700 legacy=0x7f75cc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient: _finish_hunting 0 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 1 monclient: found mon.noname-b 2024-03-29T11:05:25.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc0ac9d0 con 0x7f75cc0b0770 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient: _finish_auth 0 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.612+0000 7f75d0905640 10 monclient: _check_auth_tickets 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.613+0000 7f75d0905640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.613+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f75c80015f0 con 0x7f75cc0b0770 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.613+0000 7f75d3b6b640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.613+0000 7f75d3b6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.613+0000 7f75d0905640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.613+0000 7f75d0905640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.613+0000 7f75d0905640 10 monclient: dump: 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:25.596 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.613+0000 7f75d3b6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.615+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75c8001810 con 0x7f75cc0b0770 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.615+0000 7f75d0905640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.615+0000 7f75d0905640 1 -- 172.21.15.107:0/420707262 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f75c8001ba0 con 0x7f75cc0b0770 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.615+0000 7f75d3b6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:25.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.615+0000 7f75d3b6b640 4 set_mon_vals no callback set 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d3b6b640 10 monclient: shutdown 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d0905640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d0905640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d0905640 10 monclient: dump: 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:25.600 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d3b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d3b6b640 1 -- 172.21.15.107:0/420707262 >> v1:172.21.15.107:6790/0 conn(0x7f75cc0b0770 legacy=0x7f75cc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d3b6b640 1 -- 172.21.15.107:0/420707262 shutdown_connections 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d3b6b640 1 -- 172.21.15.107:0/420707262 >> 172.21.15.107:0/420707262 conn(0x7f75cc06b660 msgr2=0x7f75cc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d3b6b640 1 -- 172.21.15.107:0/420707262 shutdown_connections 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.616+0000 7f75d3b6b640 1 -- 172.21.15.107:0/420707262 wait complete. 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.617+0000 7f75d3b6b640 10 monclient: build_initial_monmap 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.617+0000 7f75d3b6b640 10 monclient: monmap: 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:25.601 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.617+0000 7f75d3b6b640 1 librados: starting msgr at 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.617+0000 7f75d3b6b640 1 librados: starting objecter 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.617+0000 7f75d3b6b640 1 Processor -- start 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.618+0000 7f75d3b6b640 1 -- start start 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.618+0000 7f75d3b6b640 1 librados: setting wanted keys 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.618+0000 7f75d3b6b640 1 librados: calling monclient init 2024-03-29T11:05:25.602 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.618+0000 7f75d3b6b640 10 monclient: init 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.619+0000 7f75d3b6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.619+0000 7f75d3b6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.619+0000 7f75d3b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f75cc0ac5a0 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.619+0000 7f75d3b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f75cc0b0770 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.619+0000 7f75d1907640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f75cc0ac5a0 0x7f75cc144ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54000/0 (socket says 172.21.15.107:54000) 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.620+0000 7f75d3b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f75cc14a700 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.620+0000 7f75d1907640 1 -- 172.21.15.107:0/3692959623 learned_addr learned my addr 172.21.15.107:0/3692959623 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.620+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc128360 con 0x7f75cc14a700 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.620+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc1285d0 con 0x7f75cc0b0770 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.620+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc126350 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.620+0000 7f75d3b6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.620+0000 7f75d3b6b640 10 monclient(hunting): authenticate will time out at 1256.888281s 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.620+0000 7f75c27fc640 10 client.?.objecter ms_handle_connect 0x7f75cc0ac5a0 2024-03-29T11:05:25.603 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.620+0000 7f75c27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.621+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738159441 0 0) 0x7f75cc126350 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.622+0000 7f75c27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.622+0000 7f75c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.622+0000 7f75c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.622+0000 7f75c27fc640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.622+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b0004eb0 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.622+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3136557320 0 0) 0x7f75b0004eb0 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.622+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75cc126350 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f75c80015f0 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.607 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 10 monclient(hunting): dump: 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:25.608 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 149749255 0 0) 0x7f75cc126350 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 >> v1:172.21.15.129:6789/0 conn(0x7f75cc0b0770 legacy=0x7f75cc126870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 >> v1:172.21.15.107:6789/0 conn(0x7f75cc14a700 legacy=0x7f75cc126f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 1 monclient: found mon.c 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc125f00 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 10 monclient: _finish_auth 0 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75d3b6b640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75d3b6b640 10 monclient: _renew_subs 2024-03-29T11:05:25.609 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75d3b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc1260f0 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75d3b6b640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75d3b6b640 10 monclient: _renew_subs 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75d3b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.623+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc1dfc70 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.624+0000 7f75d3b6b640 1 librados: init done 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.625+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75c8004a90 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.625+0000 7f75c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.625+0000 7f75c3fff640 4 set_mon_vals no callback set 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75d3b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f75c8004e60 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:25.610 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75c27fc640 10 monclient: dump: 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f75c8005080 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75d3b6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:25.611 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75d3b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:25.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b4005080 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.626+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f75c8084bb0 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.627+0000 7f75c27fc640 10 client.4574.objecter ms_dispatch 0x7f75cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:25.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.627+0000 7f75d1106640 10 monclient: get_auth_request con 0x7f75b006dd80 auth_method 0 2024-03-29T11:05:25.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.627+0000 7f75c27fc640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:25.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.627+0000 7f75c27fc640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:25.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.627+0000 7f75c27fc640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:05:25.612 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.627+0000 7f75c27fc640 10 client.4574.objecter ms_handle_connect 0x7f75b006dd80 2024-03-29T11:05:25.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.636+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75c8050510 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.636+0000 7f75c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:25.617 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.636+0000 7f75c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:25.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.949+0000 7f75d3b6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:25.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.949+0000 7f75d3b6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:25.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.949+0000 7f75d3b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:25.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.949+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75b4005330 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.950+0000 7f75c27fc640 1 -- 172.21.15.107:0/3692959623 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f75c8052440 con 0x7f75cc0ac5a0 2024-03-29T11:05:25.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.950+0000 7f75c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:25.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.950+0000 7f75c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:25.931 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:05:25.931 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:25.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.955+0000 7f75d3b6b640 10 librados: watch_flush enter 2024-03-29T11:05:25.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.955+0000 7f75d3b6b640 10 librados: watch_flush exit 2024-03-29T11:05:25.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.955+0000 7f75d3b6b640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:05:25.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.955+0000 7f75d3b6b640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:05:25.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.955+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 >> v1:172.21.15.107:6816/882194755 conn(0x7f75b006dd80 legacy=0x7f75b0070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:25.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.955+0000 7f75d3b6b640 10 monclient: shutdown 2024-03-29T11:05:25.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.955+0000 7f75d3b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:25.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.955+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 >> v1:172.21.15.107:6790/0 conn(0x7f75cc0ac5a0 legacy=0x7f75cc144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:25.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.957+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 shutdown_connections 2024-03-29T11:05:25.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.957+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 >> 172.21.15.107:0/3692959623 conn(0x7f75cc06b660 msgr2=0x7f75cc0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:25.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.958+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 shutdown_connections 2024-03-29T11:05:25.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.958+0000 7f75d3b6b640 1 -- 172.21.15.107:0/3692959623 wait complete. 2024-03-29T11:05:25.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:25.958+0000 7f75d3b6b640 1 librados: shutdown 2024-03-29T11:05:25.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:32.762 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:32.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.936+0000 7fb11e902640 10 monclient: get_monmap_and_config 2024-03-29T11:05:32.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.936+0000 7fb11e902640 10 monclient: build_initial_monmap 2024-03-29T11:05:32.915 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.936+0000 7fb11e902640 10 monclient: monmap: 2024-03-29T11:05:32.915 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:32.915 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:32.915 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:05:32.937080+0000 2024-03-29T11:05:32.916 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:05:32.937080+0000 2024-03-29T11:05:32.916 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:32.916 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:32.916 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:32.916 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:05:32.916 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:05:32.916 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:32.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.937+0000 7fb11e902640 1 Processor -- start 2024-03-29T11:05:32.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.937+0000 7fb11e902640 1 -- start start 2024-03-29T11:05:32.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.937+0000 7fb11e902640 10 monclient: init 2024-03-29T11:05:32.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:32.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:32.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 10 monclient(hunting): picked mon.noname-b con 0x7fb118070120 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:32.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 10 monclient(hunting): picked mon.noname-a con 0x7fb1180b0420 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:32.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 10 monclient(hunting): picked mon.noname-c con 0x7fb118070500 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:32.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118072090 con 0x7fb1180b0420 2024-03-29T11:05:32.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118072370 con 0x7fb118070500 2024-03-29T11:05:32.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118072650 con 0x7fb118070120 2024-03-29T11:05:32.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:32.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb11e902640 10 monclient(hunting): authenticate will time out at 1264.206177s 2024-03-29T11:05:32.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb117fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb118070120 0x7fb11813d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45346/0 (socket says 172.21.15.107:45346) 2024-03-29T11:05:32.918 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb117fff640 1 -- 172.21.15.107:0/4256250241 learned_addr learned my addr 172.21.15.107:0/4256250241 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:32.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580882069 0 0) 0x7fb118072090 con 0x7fb1180b0420 2024-03-29T11:05:32.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:05:32.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:32.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:32.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb104003520 con 0x7fb1180b0420 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206418974 0 0) 0x7fb118072650 con 0x7fb118070120 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.938+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb104003840 con 0x7fb118070120 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402402936 0 0) 0x7fb104003520 con 0x7fb1180b0420 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 >> v1:172.21.15.107:6790/0 conn(0x7fb118070120 legacy=0x7fb11813d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 >> v1:172.21.15.129:6789/0 conn(0x7fb118070500 legacy=0x7fb11814b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: _finish_hunting 0 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 1 monclient: found mon.noname-a 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:32.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118140090 con 0x7fb1180b0420 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: _finish_auth 0 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: _check_auth_tickets 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb100001670 con 0x7fb1180b0420 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: dump: 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:32.921 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb100001850 con 0x7fb1180b0420 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 1 -- 172.21.15.107:0/4256250241 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb100001670 con 0x7fb1180b0420 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb1177fe640 10 monclient: dump: 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.922 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb11e902640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb11e902640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.939+0000 7fb11e902640 4 set_mon_vals no callback set 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.940+0000 7fb11e902640 10 monclient: shutdown 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.940+0000 7fb11e902640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.940+0000 7fb11e902640 1 -- 172.21.15.107:0/4256250241 >> v1:172.21.15.107:6789/0 conn(0x7fb1180b0420 legacy=0x7fb1181405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.940+0000 7fb11e902640 1 -- 172.21.15.107:0/4256250241 shutdown_connections 2024-03-29T11:05:32.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.940+0000 7fb11e902640 1 -- 172.21.15.107:0/4256250241 >> 172.21.15.107:0/4256250241 conn(0x7fb118067500 msgr2=0x7fb1180b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:32.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.940+0000 7fb11e902640 1 -- 172.21.15.107:0/4256250241 shutdown_connections 2024-03-29T11:05:32.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.940+0000 7fb11e902640 1 -- 172.21.15.107:0/4256250241 wait complete. 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.941+0000 7fb11e902640 10 monclient: build_initial_monmap 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.941+0000 7fb11e902640 10 monclient: monmap: 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:32.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.941+0000 7fb11e902640 1 librados: starting msgr at 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.941+0000 7fb11e902640 1 librados: starting objecter 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.942+0000 7fb11e902640 1 Processor -- start 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.942+0000 7fb11e902640 1 -- start start 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.942+0000 7fb11e902640 1 librados: setting wanted keys 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.942+0000 7fb11e902640 1 librados: calling monclient init 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.942+0000 7fb11e902640 10 monclient: init 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 10 monclient(hunting): picked mon.noname-c con 0x7fb118070120 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 10 monclient(hunting): picked mon.noname-b con 0x7fb118070500 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 10 monclient(hunting): picked mon.noname-a con 0x7fb1180b0420 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118144cb0 con 0x7fb1180b0420 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118144f20 con 0x7fb118070500 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118143da0 con 0x7fb118070120 2024-03-29T11:05:32.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb11e902640 10 monclient(hunting): authenticate will time out at 1264.211390s 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb117fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb118070120 0x7fb118138da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:45580/0 (socket says 172.21.15.107:45580) 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb117fff640 1 -- 172.21.15.107:0/1441115155 learned_addr learned my addr 172.21.15.107:0/1441115155 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 10 client.?.objecter ms_handle_connect 0x7fb1180b0420 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 10 client.?.objecter ms_handle_connect 0x7fb118070120 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618316736 0 0) 0x7fb118143da0 con 0x7fb118070120 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 10 monclient(hunting): _negotiate 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.943+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f8004eb0 con 0x7fb118070120 2024-03-29T11:05:32.930 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 client.?.objecter ms_handle_connect 0x7fb118070500 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142186950 0 0) 0x7fb118144f20 con 0x7fb118070500 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): _negotiate 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb118143da0 con 0x7fb118070500 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487362601 0 0) 0x7fb118144cb0 con 0x7fb1180b0420 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): _negotiate 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb118144f20 con 0x7fb1180b0420 2024-03-29T11:05:32.931 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1622222293 0 0) 0x7fb0f8004eb0 con 0x7fb118070120 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb118144cb0 con 0x7fb118070120 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb108001930 con 0x7fb118070120 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): dump: 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:32.932 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1981809745 0 0) 0x7fb118144f20 con 0x7fb1180b0420 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0f8004eb0 con 0x7fb1180b0420 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb110001b00 con 0x7fb1180b0420 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.944+0000 7fb1157fa640 10 monclient(hunting): dump: 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:32.933 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2794466526 0 0) 0x7fb0f8004eb0 con 0x7fb1180b0420 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 >> v1:172.21.15.107:6790/0 conn(0x7fb118070120 legacy=0x7fb118138da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 >> v1:172.21.15.129:6789/0 conn(0x7fb118070500 legacy=0x7fb1181d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 10 monclient: _finish_hunting 0 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 1 monclient: found mon.a 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:32.934 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118143880 con 0x7fb1180b0420 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 10 monclient: _finish_auth 0 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 10 monclient: _check_auth_tickets 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb11e902640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb11e902640 10 monclient: _renew_subs 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb11e902640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb118143010 con 0x7fb1180b0420 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb11e902640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb11e902640 10 monclient: _renew_subs 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb11e902640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1181435d0 con 0x7fb1180b0420 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb110002030 con 0x7fb1180b0420 2024-03-29T11:05:32.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb110003b10 con 0x7fb1180b0420 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb116ffd640 4 set_mon_vals no callback set 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb1157fa640 10 monclient: dump: 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:32.936 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.945+0000 7fb11e902640 1 librados: init done 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.947+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb110019ae0 con 0x7fb1180b0420 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fb110083a50 con 0x7fb1180b0420 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb1157fa640 10 client.4503.objecter ms_dispatch 0x7fb118000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb1157fa640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb1157fa640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb1157fa640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb117fff640 10 monclient: get_auth_request con 0x7fb0f806e000 auth_method 0 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb11e902640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb11e902640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb1157fa640 10 client.4503.objecter ms_handle_connect 0x7fb0f806e000 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb11e902640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:32.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.948+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1181431a0 con 0x7fb1180b0420 2024-03-29T11:05:32.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.956+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb11004f160 con 0x7fb1180b0420 2024-03-29T11:05:32.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.956+0000 7fb1157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:32.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:32.956+0000 7fb1157fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:33.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.307+0000 7fb11e902640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:33.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.307+0000 7fb11e902640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:33.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.307+0000 7fb11e902640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:33.286 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.307+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb118075f10 con 0x7fb1180b0420 2024-03-29T11:05:33.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.309+0000 7fb1157fa640 1 -- 172.21.15.107:0/1441115155 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fb110051260 con 0x7fb1180b0420 2024-03-29T11:05:33.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.309+0000 7fb1157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:33.288 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.309+0000 7fb1157fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:33.289 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:05:33.289 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:33.291 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.312+0000 7fb11e902640 10 librados: watch_flush enter 2024-03-29T11:05:33.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.312+0000 7fb11e902640 10 librados: watch_flush exit 2024-03-29T11:05:33.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.312+0000 7fb11e902640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-29T11:05:33.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.312+0000 7fb11e902640 10 client.4503.objecter successfully canceled tick 2024-03-29T11:05:33.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.312+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 >> v1:172.21.15.107:6816/882194755 conn(0x7fb0f806e000 legacy=0x7fb0f80704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:33.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.312+0000 7fb11e902640 10 monclient: shutdown 2024-03-29T11:05:33.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.312+0000 7fb11e902640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:33.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.312+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 >> v1:172.21.15.107:6789/0 conn(0x7fb1180b0420 legacy=0x7fb1181e2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:33.292 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.313+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 shutdown_connections 2024-03-29T11:05:33.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.313+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 >> 172.21.15.107:0/1441115155 conn(0x7fb118067500 msgr2=0x7fb11813a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:33.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.313+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 shutdown_connections 2024-03-29T11:05:33.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.313+0000 7fb11e902640 1 -- 172.21.15.107:0/1441115155 wait complete. 2024-03-29T11:05:33.293 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:33.313+0000 7fb11e902640 1 librados: shutdown 2024-03-29T11:05:33.309 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:40.116 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.273+0000 7f87220fb640 10 monclient: get_monmap_and_config 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.273+0000 7f87220fb640 10 monclient: build_initial_monmap 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.273+0000 7f87220fb640 10 monclient: monmap: 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:05:40.274713+0000 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:05:40.274713+0000 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:40.253 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:40.254 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:05:40.254 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:05:40.254 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:40.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.274+0000 7f87220fb640 1 Processor -- start 2024-03-29T11:05:40.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.274+0000 7f87220fb640 1 -- start start 2024-03-29T11:05:40.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.274+0000 7f87220fb640 10 monclient: init 2024-03-29T11:05:40.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:40.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:05:40.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 10 monclient(hunting): picked mon.noname-c con 0x7f871c1422b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:40.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 10 monclient(hunting): picked mon.noname-b con 0x7f871c142690 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:40.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 10 monclient(hunting): picked mon.noname-a con 0x7f871c0b17d0 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:40.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c142c80 con 0x7f871c0b17d0 2024-03-29T11:05:40.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c0aae40 con 0x7f871c1422b0 2024-03-29T11:05:40.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c13fbd0 con 0x7f871c142690 2024-03-29T11:05:40.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:40.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.276+0000 7f87220fb640 10 monclient(hunting): authenticate will time out at 1271.544118s 2024-03-29T11:05:40.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.278+0000 7f871bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f871c0b17d0 0x7f871c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46241/0 (socket says 172.21.15.107:46241) 2024-03-29T11:05:40.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.278+0000 7f871bfff640 1 -- 172.21.15.107:0/4165969014 learned_addr learned my addr 172.21.15.107:0/4165969014 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:40.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871affd640 1 --1- 172.21.15.107:0/4165969014 >> v1:172.21.15.107:6790/0 conn(0x7f871c142690 0x7f871c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41386/0 (socket says 172.21.15.107:41386) 2024-03-29T11:05:40.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 1 -- 172.21.15.107:0/4165969014 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665846261 0 0) 0x7f871c142c80 con 0x7f871c0b17d0 2024-03-29T11:05:40.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:40.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:40.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:40.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:05:40.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 1 -- 172.21.15.107:0/4165969014 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8704003520 con 0x7f871c0b17d0 2024-03-29T11:05:40.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 1 -- 172.21.15.107:0/4165969014 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478851209 0 0) 0x7f8704003520 con 0x7f871c0b17d0 2024-03-29T11:05:40.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 1 -- 172.21.15.107:0/4165969014 >> v1:172.21.15.107:6790/0 conn(0x7f871c142690 legacy=0x7f871c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:40.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 1 -- 172.21.15.107:0/4165969014 >> v1:172.21.15.129:6789/0 conn(0x7f871c1422b0 legacy=0x7f871c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:05:40.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:05:40.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:05:40.262 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.279+0000 7f871a7fc640 1 -- 172.21.15.107:0/4165969014 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c151940 con 0x7f871c0b17d0 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 1 -- 172.21.15.107:0/4165969014 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f8714001a10 con 0x7f871c0b17d0 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 10 monclient: dump: 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:40.263 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 1 -- 172.21.15.107:0/4165969014 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8714001c30 con 0x7f871c0b17d0 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 1 -- 172.21.15.107:0/4165969014 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f8714001fc0 con 0x7f871c0b17d0 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f871a7fc640 10 monclient: dump: 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:40.264 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.280+0000 7f87220fb640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.281+0000 7f87220fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.281+0000 7f87220fb640 4 set_mon_vals no callback set 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.281+0000 7f87220fb640 10 monclient: shutdown 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.282+0000 7f87220fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.282+0000 7f87220fb640 1 -- 172.21.15.107:0/4165969014 >> v1:172.21.15.107:6789/0 conn(0x7f871c0b17d0 legacy=0x7f871c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.282+0000 7f87220fb640 1 -- 172.21.15.107:0/4165969014 shutdown_connections 2024-03-29T11:05:40.265 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.282+0000 7f87220fb640 1 -- 172.21.15.107:0/4165969014 >> 172.21.15.107:0/4165969014 conn(0x7f871c066a00 msgr2=0x7f871c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:40.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.282+0000 7f87220fb640 1 -- 172.21.15.107:0/4165969014 shutdown_connections 2024-03-29T11:05:40.266 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.283+0000 7f87220fb640 1 -- 172.21.15.107:0/4165969014 wait complete. 2024-03-29T11:05:40.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.284+0000 7f87220fb640 10 monclient: build_initial_monmap 2024-03-29T11:05:40.267 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.284+0000 7f87220fb640 10 monclient: monmap: 2024-03-29T11:05:40.267 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:40.267 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:40.267 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:05:40.267 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.284+0000 7f87220fb640 1 librados: starting msgr at 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.284+0000 7f87220fb640 1 librados: starting objecter 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.284+0000 7f87220fb640 1 Processor -- start 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.284+0000 7f87220fb640 1 -- start start 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.284+0000 7f87220fb640 1 librados: setting wanted keys 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.284+0000 7f87220fb640 1 librados: calling monclient init 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.284+0000 7f87220fb640 10 monclient: init 2024-03-29T11:05:40.268 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.285+0000 7f87220fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.285+0000 7f87220fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.285+0000 7f87220fb640 10 monclient(hunting): picked mon.noname-b con 0x7f871c0b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.285+0000 7f87220fb640 10 monclient(hunting): picked mon.noname-c con 0x7f871c1422b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.286+0000 7f871affd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f871c1422b0 0x7f871c12bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:41454/0 (socket says 172.21.15.107:41454) 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.286+0000 7f871affd640 1 -- 172.21.15.107:0/2266827052 learned_addr learned my addr 172.21.15.107:0/2266827052 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.286+0000 7f87220fb640 10 monclient(hunting): picked mon.noname-a con 0x7f871c142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.286+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c06e760 con 0x7f871c142690 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.286+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c06f260 con 0x7f871c0b17d0 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.286+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c06f540 con 0x7f871c1422b0 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.286+0000 7f87220fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:40.269 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.286+0000 7f87220fb640 10 monclient(hunting): authenticate will time out at 1271.554410s 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 client.?.objecter ms_handle_connect 0x7f871c1422b0 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394546508 0 0) 0x7f871c06f540 con 0x7f871c1422b0 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8708004eb0 con 0x7f871c1422b0 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2941441220 0 0) 0x7f8708004eb0 con 0x7f871c1422b0 2024-03-29T11:05:40.275 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f871c06f540 con 0x7f871c1422b0 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f870c001930 con 0x7f871c1422b0 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 monclient(hunting): dump: 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:40.276 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.287+0000 7f8703fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.288+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 227376109 0 0) 0x7f871c06f540 con 0x7f871c1422b0 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.288+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 >> v1:172.21.15.129:6789/0 conn(0x7f871c0b17d0 legacy=0x7f871c06e050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 >> v1:172.21.15.107:6789/0 conn(0x7f871c142690 legacy=0x7f871c12c320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f8703fff640 10 monclient: _finish_hunting 0 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f8703fff640 1 monclient: found mon.c 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f8703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f8703fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c12e770 con 0x7f871c1422b0 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f8703fff640 10 monclient: _finish_auth 0 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f8703fff640 10 monclient: _check_auth_tickets 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f8703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:40.277 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f87220fb640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f87220fb640 10 monclient: _renew_subs 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f87220fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c12e200 con 0x7f871c1422b0 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f87220fb640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f87220fb640 10 monclient: _renew_subs 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f87220fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c1dec20 con 0x7f871c1422b0 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.289+0000 7f87220fb640 1 librados: init done 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.290+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f870c001c50 con 0x7f871c1422b0 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.290+0000 7f8703fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.290+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f870c004c30 con 0x7f871c1422b0 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.290+0000 7f8703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.290+0000 7f8703fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:40.278 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.290+0000 7f87220fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.290+0000 7f8703fff640 10 monclient: dump: 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.290+0000 7f8719ffb640 4 set_mon_vals no callback set 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.291+0000 7f87220fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:40.279 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.291+0000 7f87220fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:40.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.291+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871c075f10 con 0x7f871c1422b0 2024-03-29T11:05:40.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.291+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f870c004e50 con 0x7f871c1422b0 2024-03-29T11:05:40.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.292+0000 7f871b7fe640 10 monclient: get_auth_request con 0x7f870806dcd0 auth_method 0 2024-03-29T11:05:40.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.293+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f870c084b70 con 0x7f871c1422b0 2024-03-29T11:05:40.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.293+0000 7f8703fff640 10 client.4586.objecter ms_dispatch 0x7f871c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:40.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.293+0000 7f8703fff640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:40.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.293+0000 7f8703fff640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:40.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.293+0000 7f8703fff640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:05:40.280 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.293+0000 7f8703fff640 10 client.4586.objecter ms_handle_connect 0x7f870806dcd0 2024-03-29T11:05:40.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.299+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f870c084e60 con 0x7f871c1422b0 2024-03-29T11:05:40.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.299+0000 7f8703fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:40.281 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.299+0000 7f8703fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:40.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.608+0000 7f87220fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:40.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.608+0000 7f87220fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:40.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.608+0000 7f87220fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:40.587 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.608+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f871c12e550 con 0x7f871c1422b0 2024-03-29T11:05:40.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.609+0000 7f8703fff640 1 -- 172.21.15.107:0/2266827052 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f870c050450 con 0x7f871c1422b0 2024-03-29T11:05:40.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.609+0000 7f8703fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:40.588 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.609+0000 7f8703fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:40.588 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:05:40.588 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:40.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.612+0000 7f87220fb640 10 librados: watch_flush enter 2024-03-29T11:05:40.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.612+0000 7f87220fb640 10 librados: watch_flush exit 2024-03-29T11:05:40.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.612+0000 7f87220fb640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:05:40.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.612+0000 7f87220fb640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:05:40.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.612+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 >> v1:172.21.15.107:6816/882194755 conn(0x7f870806dcd0 legacy=0x7f8708070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:40.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.612+0000 7f87220fb640 10 monclient: shutdown 2024-03-29T11:05:40.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.612+0000 7f87220fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:40.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.612+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 >> v1:172.21.15.107:6790/0 conn(0x7f871c1422b0 legacy=0x7f871c12bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:40.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.613+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 shutdown_connections 2024-03-29T11:05:40.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.613+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 >> 172.21.15.107:0/2266827052 conn(0x7f871c066a00 msgr2=0x7f871c13f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:40.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.613+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 shutdown_connections 2024-03-29T11:05:40.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.613+0000 7f87220fb640 1 -- 172.21.15.107:0/2266827052 wait complete. 2024-03-29T11:05:40.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:40.613+0000 7f87220fb640 1 librados: shutdown 2024-03-29T11:05:40.606 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:47.412 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:47.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.586+0000 7f794000f640 10 monclient: get_monmap_and_config 2024-03-29T11:05:47.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.586+0000 7f794000f640 10 monclient: build_initial_monmap 2024-03-29T11:05:47.567 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.587+0000 7f794000f640 10 monclient: monmap: 2024-03-29T11:05:47.567 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:47.567 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:05:47.587954+0000 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:05:47.587954+0000 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.588+0000 7f794000f640 1 Processor -- start 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.588+0000 7f794000f640 1 -- start start 2024-03-29T11:05:47.568 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.588+0000 7f794000f640 10 monclient: init 2024-03-29T11:05:47.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:47.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:05:47.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 10 monclient(hunting): picked mon.noname-a con 0x7f79380b4100 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:47.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 10 monclient(hunting): picked mon.noname-b con 0x7f793813ee90 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:47.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 10 monclient(hunting): picked mon.noname-c con 0x7f79380af900 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:47.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79380afd10 con 0x7f79380b4100 2024-03-29T11:05:47.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79380af6f0 con 0x7f79380af900 2024-03-29T11:05:47.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79380aaf60 con 0x7f793813ee90 2024-03-29T11:05:47.569 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f794000f640 10 monclient(hunting): authenticate will time out at 1278.857475s 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f793d5aa640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f793813ee90 0x7f793813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40612/0 (socket says 172.21.15.107:40612) 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f793d5aa640 1 -- 172.21.15.107:0/638489847 learned_addr learned my addr 172.21.15.107:0/638489847 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f793cda9640 1 -- 172.21.15.107:0/638489847 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840100450 0 0) 0x7f79380aaf60 con 0x7f793813ee90 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f793cda9640 10 monclient(hunting): _negotiate 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f793cda9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f793cda9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f793cda9640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.589+0000 7f793cda9640 1 -- 172.21.15.107:0/638489847 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7920003520 con 0x7f793813ee90 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.590+0000 7f793cda9640 1 -- 172.21.15.107:0/638489847 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3486388396 0 0) 0x7f7920003520 con 0x7f793813ee90 2024-03-29T11:05:47.570 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.590+0000 7f793cda9640 1 -- 172.21.15.107:0/638489847 >> v1:172.21.15.129:6789/0 conn(0x7f79380af900 legacy=0x7f7938149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:05:47.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 1 -- 172.21.15.107:0/638489847 >> v1:172.21.15.107:6789/0 conn(0x7f79380b4100 legacy=0x7f79380a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:05:47.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 10 monclient: _finish_hunting 0 2024-03-29T11:05:47.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 1 monclient: found mon.noname-b 2024-03-29T11:05:47.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:47.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:05:47.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 1 -- 172.21.15.107:0/638489847 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793813e970 con 0x7f793813ee90 2024-03-29T11:05:47.573 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 10 monclient: _finish_auth 0 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 10 monclient: _check_auth_tickets 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f794000f640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f794000f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 1 -- 172.21.15.107:0/638489847 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f792c001690 con 0x7f793813ee90 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f793cda9640 10 monclient: dump: 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:47.574 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:47.575 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:47.575 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:47.575 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:47.575 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:47.575 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:47.575 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.593+0000 7f794000f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.594+0000 7f793cda9640 1 -- 172.21.15.107:0/638489847 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f792c001920 con 0x7f793813ee90 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.594+0000 7f793cda9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.594+0000 7f794000f640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.594+0000 7f794000f640 4 set_mon_vals no callback set 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.595+0000 7f794000f640 10 monclient: shutdown 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.595+0000 7f794000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.595+0000 7f794000f640 1 -- 172.21.15.107:0/638489847 >> v1:172.21.15.107:6790/0 conn(0x7f793813ee90 legacy=0x7f793813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.595+0000 7f794000f640 1 -- 172.21.15.107:0/638489847 shutdown_connections 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.595+0000 7f794000f640 1 -- 172.21.15.107:0/638489847 >> 172.21.15.107:0/638489847 conn(0x7f79380b35e0 msgr2=0x7f79380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.596+0000 7f794000f640 1 -- 172.21.15.107:0/638489847 shutdown_connections 2024-03-29T11:05:47.576 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.596+0000 7f794000f640 1 -- 172.21.15.107:0/638489847 wait complete. 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.597+0000 7f794000f640 10 monclient: build_initial_monmap 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.597+0000 7f794000f640 10 monclient: monmap: 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:05:47.577 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:47.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.598+0000 7f794000f640 1 librados: starting msgr at 2024-03-29T11:05:47.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.598+0000 7f794000f640 1 librados: starting objecter 2024-03-29T11:05:47.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.598+0000 7f794000f640 1 Processor -- start 2024-03-29T11:05:47.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.598+0000 7f794000f640 1 -- start start 2024-03-29T11:05:47.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.598+0000 7f794000f640 1 librados: setting wanted keys 2024-03-29T11:05:47.578 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.598+0000 7f794000f640 1 librados: calling monclient init 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.598+0000 7f794000f640 10 monclient: init 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 10 monclient(hunting): picked mon.noname-b con 0x7f79380af900 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 10 monclient(hunting): picked mon.noname-a con 0x7f79380b4100 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 10 monclient(hunting): picked mon.noname-c con 0x7f793813ee90 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7938070460 con 0x7f79380b4100 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793813a1c0 con 0x7f79380af900 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793813a350 con 0x7f793813ee90 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:47.579 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.599+0000 7f794000f640 10 monclient(hunting): authenticate will time out at 1278.867458s 2024-03-29T11:05:47.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.601+0000 7f793e5ac640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f793813ee90 0x7f7938139ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40784/0 (socket says 172.21.15.107:40784) 2024-03-29T11:05:47.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.602+0000 7f793e5ac640 1 -- 172.21.15.107:0/2205595714 learned_addr learned my addr 172.21.15.107:0/2205595714 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:47.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.602+0000 7f79367fc640 10 client.?.objecter ms_handle_connect 0x7f793813ee90 2024-03-29T11:05:47.581 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.602+0000 7f79367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:47.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.603+0000 7f79367fc640 10 client.?.objecter ms_handle_connect 0x7f79380b4100 2024-03-29T11:05:47.582 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.603+0000 7f79367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934268639 0 0) 0x7f7938070460 con 0x7f79380b4100 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7924004eb0 con 0x7f79380b4100 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065169330 0 0) 0x7f793813a350 con 0x7f793813ee90 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7938070460 con 0x7f793813ee90 2024-03-29T11:05:47.584 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 client.?.objecter ms_handle_connect 0x7f79380af900 2024-03-29T11:05:47.585 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.604+0000 7f79367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:47.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198983847 0 0) 0x7f793813a1c0 con 0x7f79380af900 2024-03-29T11:05:47.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:47.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:47.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:47.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:05:47.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f793813a350 con 0x7f79380af900 2024-03-29T11:05:47.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2019621819 0 0) 0x7f7938070460 con 0x7f793813ee90 2024-03-29T11:05:47.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7924006680 con 0x7f793813ee90 2024-03-29T11:05:47.589 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7930001b30 con 0x7f793813ee90 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 10 monclient(hunting): dump: 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:47.590 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.605+0000 7f79367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2177400478 0 0) 0x7f7924006680 con 0x7f793813ee90 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 >> v1:172.21.15.129:6789/0 conn(0x7f79380af900 legacy=0x7f79381e3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 >> v1:172.21.15.107:6789/0 conn(0x7f79380b4100 legacy=0x7f7938070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 1 monclient: found mon.c 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793806f940 con 0x7f793813ee90 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 10 monclient: _finish_auth 0 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f79367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f794000f640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:05:47.591 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f794000f640 10 monclient: _renew_subs 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f794000f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f794000f640 1 -- 172.21.15.107:0/2205595714 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7938072430 con 0x7f793813ee90 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f794000f640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f794000f640 10 monclient: _renew_subs 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f794000f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f794000f640 1 -- 172.21.15.107:0/2205595714 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f793814d910 con 0x7f793813ee90 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.606+0000 7f794000f640 1 librados: init done 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f791ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f791ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7930002040 con 0x7f793813ee90 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f791ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f791ffff640 1 -- 172.21.15.107:0/2205595714 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793806fbb0 con 0x7f793813ee90 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f79367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f7930001b30 con 0x7f793813ee90 2024-03-29T11:05:47.592 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f79367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f79367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.608+0000 7f79367fc640 10 monclient: dump: 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:47.593 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:47.594 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.610+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f793001aaf0 con 0x7f793813ee90 2024-03-29T11:05:47.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.610+0000 7f7937fff640 4 set_mon_vals no callback set 2024-03-29T11:05:47.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.611+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f79300547a0 con 0x7f793813ee90 2024-03-29T11:05:47.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.611+0000 7f79367fc640 10 client.4592.objecter ms_dispatch 0x7f7938000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:47.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.611+0000 7f79367fc640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:47.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.611+0000 7f79367fc640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:47.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.611+0000 7f79367fc640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:05:47.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.611+0000 7f793ddab640 10 monclient: get_auth_request con 0x7f792406e1a0 auth_method 0 2024-03-29T11:05:47.595 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.612+0000 7f79367fc640 10 client.4592.objecter ms_handle_connect 0x7f792406e1a0 2024-03-29T11:05:47.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.617+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7930050a40 con 0x7f793813ee90 2024-03-29T11:05:47.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.617+0000 7f79367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:47.597 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.617+0000 7f79367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:47.949 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.970+0000 7f791ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:47.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.970+0000 7f791ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:47.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.970+0000 7f791ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:05:47.950 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.970+0000 7f791ffff640 1 -- 172.21.15.107:0/2205595714 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79380aec30 con 0x7f793813ee90 2024-03-29T11:05:47.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.972+0000 7f79367fc640 1 -- 172.21.15.107:0/2205595714 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f7930050be0 con 0x7f793813ee90 2024-03-29T11:05:47.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.972+0000 7f79367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:47.951 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.972+0000 7f79367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:47.951 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:05:47.952 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.975+0000 7f791ffff640 10 librados: watch_flush enter 2024-03-29T11:05:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 10 librados: watch_flush exit 2024-03-29T11:05:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:05:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:05:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 1 -- 172.21.15.107:0/2205595714 >> v1:172.21.15.107:6816/882194755 conn(0x7f792406e1a0 legacy=0x7f7924070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 10 monclient: shutdown 2024-03-29T11:05:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:47.955 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 1 -- 172.21.15.107:0/2205595714 >> v1:172.21.15.107:6790/0 conn(0x7f793813ee90 legacy=0x7f7938139ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:47.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 1 -- 172.21.15.107:0/2205595714 shutdown_connections 2024-03-29T11:05:47.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 1 -- 172.21.15.107:0/2205595714 >> 172.21.15.107:0/2205595714 conn(0x7f79380b35e0 msgr2=0x7f79380b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:47.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.976+0000 7f791ffff640 1 -- 172.21.15.107:0/2205595714 shutdown_connections 2024-03-29T11:05:47.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.977+0000 7f791ffff640 1 -- 172.21.15.107:0/2205595714 wait complete. 2024-03-29T11:05:47.956 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:47.977+0000 7f791ffff640 1 librados: shutdown 2024-03-29T11:05:47.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:54.780 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:54.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.936+0000 7ff9518ba640 10 monclient: get_monmap_and_config 2024-03-29T11:05:54.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.936+0000 7ff9518ba640 10 monclient: build_initial_monmap 2024-03-29T11:05:54.916 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.936+0000 7ff9518ba640 10 monclient: monmap: 2024-03-29T11:05:54.916 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:54.916 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:54.916 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:05:54.937357+0000 2024-03-29T11:05:54.916 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:05:54.937357+0000 2024-03-29T11:05:54.916 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:54.917 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:54.917 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:54.917 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:05:54.917 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:05:54.917 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:54.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.937+0000 7ff9518ba640 1 Processor -- start 2024-03-29T11:05:54.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.937+0000 7ff9518ba640 1 -- start start 2024-03-29T11:05:54.917 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.937+0000 7ff9518ba640 10 monclient: init 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.937+0000 7ff9518ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.937+0000 7ff9518ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff9518ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff94c0b0770 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff9518ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff94c0ad850 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff9518ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff94c13c1b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff9518ba640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c0ac8e0 con 0x7ff94c0ad850 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff9518ba640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c0ad330 con 0x7ff94c0b0770 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff94a7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff94c0ad850 0x7ff94c0aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38184/0 (socket says 172.21.15.107:38184) 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff94a7fc640 1 -- 172.21.15.107:0/2605754909 learned_addr learned my addr 172.21.15.107:0/2605754909 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff9518ba640 1 -- 172.21.15.107:0/2605754909 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c0acfe0 con 0x7ff94c13c1b0 2024-03-29T11:05:54.919 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff9518ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:54.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff9518ba640 10 monclient(hunting): authenticate will time out at 1286.206278s 2024-03-29T11:05:54.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff949ffb640 1 -- 172.21.15.107:0/2605754909 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925615707 0 0) 0x7ff94c0acfe0 con 0x7ff94c13c1b0 2024-03-29T11:05:54.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff949ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:05:54.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff949ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:54.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:54.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff949ffb640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:05:54.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff949ffb640 1 -- 172.21.15.107:0/2605754909 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff934005730 con 0x7ff94c13c1b0 2024-03-29T11:05:54.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff949ffb640 1 -- 172.21.15.107:0/2605754909 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4088708590 0 0) 0x7ff934005730 con 0x7ff94c13c1b0 2024-03-29T11:05:54.920 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.938+0000 7ff949ffb640 1 -- 172.21.15.107:0/2605754909 >> v1:172.21.15.129:6789/0 conn(0x7ff94c0b0770 legacy=0x7ff94c072070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:05:54.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 1 -- 172.21.15.107:0/2605754909 >> v1:172.21.15.107:6789/0 conn(0x7ff94c0ad850 legacy=0x7ff94c0aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:54.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:05:54.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 1 monclient: found mon.noname-b 2024-03-29T11:05:54.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:54.921 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 1 -- 172.21.15.107:0/2605754909 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c0ab1f0 con 0x7ff94c13c1b0 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: _finish_auth 0 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 1 -- 172.21.15.107:0/2605754909 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff944001a10 con 0x7ff94c13c1b0 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: dump: 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:54.922 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff9518ba640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff9518ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 1 -- 172.21.15.107:0/2605754909 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff944001be0 con 0x7ff94c13c1b0 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 1 -- 172.21.15.107:0/2605754909 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff944002030 con 0x7ff94c13c1b0 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:54.923 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff949ffb640 10 monclient: dump: 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff9518ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.941+0000 7ff9518ba640 4 set_mon_vals no callback set 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.942+0000 7ff9518ba640 10 monclient: shutdown 2024-03-29T11:05:54.924 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.942+0000 7ff9518ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.942+0000 7ff9518ba640 1 -- 172.21.15.107:0/2605754909 >> v1:172.21.15.107:6790/0 conn(0x7ff94c13c1b0 legacy=0x7ff94c14a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.942+0000 7ff9518ba640 1 -- 172.21.15.107:0/2605754909 shutdown_connections 2024-03-29T11:05:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.942+0000 7ff9518ba640 1 -- 172.21.15.107:0/2605754909 >> 172.21.15.107:0/2605754909 conn(0x7ff94c06b660 msgr2=0x7ff94c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.942+0000 7ff9518ba640 1 -- 172.21.15.107:0/2605754909 shutdown_connections 2024-03-29T11:05:54.925 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.942+0000 7ff9518ba640 1 -- 172.21.15.107:0/2605754909 wait complete. 2024-03-29T11:05:54.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.943+0000 7ff9518ba640 10 monclient: build_initial_monmap 2024-03-29T11:05:54.927 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.943+0000 7ff9518ba640 10 monclient: monmap: 2024-03-29T11:05:54.927 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:05:54.927 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:54.927 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:05:54.927 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:05:54.927 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.943+0000 7ff9518ba640 1 librados: starting msgr at 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.943+0000 7ff9518ba640 1 librados: starting objecter 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.944+0000 7ff9518ba640 1 Processor -- start 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.944+0000 7ff9518ba640 1 -- start start 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.944+0000 7ff9518ba640 1 librados: setting wanted keys 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.944+0000 7ff9518ba640 1 librados: calling monclient init 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.944+0000 7ff9518ba640 10 monclient: init 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:05:54.928 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff94c0ad850 addr v1:172.21.15.107:6789/0 2024-03-29T11:05:54.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff94c0b0770 addr v1:172.21.15.107:6790/0 2024-03-29T11:05:54.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff94c13c1b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:05:54.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c138630 con 0x7ff94c0ad850 2024-03-29T11:05:54.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c13fe80 con 0x7ff94c13c1b0 2024-03-29T11:05:54.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c1395e0 con 0x7ff94c0b0770 2024-03-29T11:05:54.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:54.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff9518ba640 10 monclient(hunting): authenticate will time out at 1286.213178s 2024-03-29T11:05:54.929 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff94affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff94c0ad850 0x7ff94c137ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38302/0 (socket says 172.21.15.107:38302) 2024-03-29T11:05:54.935 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff94affd640 1 -- 172.21.15.107:0/2715656161 learned_addr learned my addr 172.21.15.107:0/2715656161 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 client.?.objecter ms_handle_connect 0x7ff94c0ad850 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146619416 0 0) 0x7ff94c138630 con 0x7ff94c0ad850 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff938004eb0 con 0x7ff94c0ad850 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 973282858 0 0) 0x7ff938004eb0 con 0x7ff94c0ad850 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff94c138630 con 0x7ff94c0ad850 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff93c001870 con 0x7ff94c0ad850 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:54.936 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 monclient(hunting): dump: 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3824953443 0 0) 0x7ff94c138630 con 0x7ff94c0ad850 2024-03-29T11:05:54.937 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 >> v1:172.21.15.107:6790/0 conn(0x7ff94c0b0770 legacy=0x7ff94c138ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:54.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.945+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 >> v1:172.21.15.129:6789/0 conn(0x7ff94c13c1b0 legacy=0x7ff94c1367c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:54.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 10 monclient: _finish_hunting 0 2024-03-29T11:05:54.938 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 1 monclient: found mon.a 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c139770 con 0x7ff94c0ad850 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 10 monclient: _finish_auth 0 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 10 monclient: _check_auth_tickets 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9518ba640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9518ba640 10 monclient: _renew_subs 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9518ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c0b2140 con 0x7ff94c0ad850 2024-03-29T11:05:54.939 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff93c004900 con 0x7ff94c0ad850 2024-03-29T11:05:54.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:54.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9518ba640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:54.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff93c004a70 con 0x7ff94c0ad850 2024-03-29T11:05:54.940 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff933fff640 10 monclient: dump: 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9518ba640 10 monclient: _renew_subs 2024-03-29T11:05:54.941 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9518ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:54.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c0b24e0 con 0x7ff94c0ad850 2024-03-29T11:05:54.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9518ba640 1 librados: init done 2024-03-29T11:05:54.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.948+0000 7ff9497fa640 4 set_mon_vals no callback set 2024-03-29T11:05:54.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.950+0000 7ff9518ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:54.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.950+0000 7ff9518ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:54.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.950+0000 7ff9518ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:54.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.950+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c075f10 con 0x7ff94c0ad850 2024-03-29T11:05:54.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.951+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7ff93c004bd0 con 0x7ff94c0ad850 2024-03-29T11:05:54.942 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.952+0000 7ff94a7fc640 10 monclient: get_auth_request con 0x7ff93806dbc0 auth_method 0 2024-03-29T11:05:54.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.952+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7ff93c0849e0 con 0x7ff94c0ad850 2024-03-29T11:05:54.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.952+0000 7ff933fff640 10 client.4515.objecter ms_dispatch 0x7ff94c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:54.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.952+0000 7ff933fff640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:54.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.952+0000 7ff933fff640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:54.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.952+0000 7ff933fff640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T11:05:54.943 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.953+0000 7ff933fff640 10 client.4515.objecter ms_handle_connect 0x7ff93806dbc0 2024-03-29T11:05:54.944 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.958+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff93c050320 con 0x7ff94c0ad850 2024-03-29T11:05:54.944 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.958+0000 7ff933fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:54.944 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:54.958+0000 7ff933fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:55.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.309+0000 7ff9518ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:55.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.309+0000 7ff9518ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:55.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.309+0000 7ff9518ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:05:55.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.309+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff94c0b22d0 con 0x7ff94c0ad850 2024-03-29T11:05:55.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.310+0000 7ff933fff640 1 -- 172.21.15.107:0/2715656161 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7ff93c084090 con 0x7ff94c0ad850 2024-03-29T11:05:55.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.310+0000 7ff933fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:55.289 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.310+0000 7ff933fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:55.290 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:05:55.290 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:55.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 10 librados: watch_flush enter 2024-03-29T11:05:55.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 10 librados: watch_flush exit 2024-03-29T11:05:55.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T11:05:55.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 10 client.4515.objecter successfully canceled tick 2024-03-29T11:05:55.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 >> v1:172.21.15.107:6816/882194755 conn(0x7ff93806dbc0 legacy=0x7ff938070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:55.294 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 10 monclient: shutdown 2024-03-29T11:05:55.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:55.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 >> v1:172.21.15.107:6789/0 conn(0x7ff94c0ad850 legacy=0x7ff94c137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:55.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 shutdown_connections 2024-03-29T11:05:55.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.314+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 >> 172.21.15.107:0/2715656161 conn(0x7ff94c06b660 msgr2=0x7ff94c135890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:55.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.315+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 shutdown_connections 2024-03-29T11:05:55.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.315+0000 7ff9518ba640 1 -- 172.21.15.107:0/2715656161 wait complete. 2024-03-29T11:05:55.295 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:05:55.315+0000 7ff9518ba640 1 librados: shutdown 2024-03-29T11:05:55.310 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:02.116 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:02.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.257+0000 7f05a45db640 10 monclient: get_monmap_and_config 2024-03-29T11:06:02.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.257+0000 7f05a45db640 10 monclient: build_initial_monmap 2024-03-29T11:06:02.237 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.258+0000 7f05a45db640 10 monclient: monmap: 2024-03-29T11:06:02.237 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:02.237 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:02.237 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:06:02.258937+0000 2024-03-29T11:06:02.237 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:06:02.258937+0000 2024-03-29T11:06:02.237 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:02.237 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:02.238 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:02.238 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:06:02.238 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:06:02.238 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:02.239 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.258+0000 7f05a45db640 1 Processor -- start 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.258+0000 7f05a45db640 1 -- start start 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.258+0000 7f05a45db640 10 monclient: init 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 10 monclient(hunting): picked mon.noname-b con 0x7f059c0b4330 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 10 monclient(hunting): picked mon.noname-a con 0x7f059c1470b0 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 10 monclient(hunting): picked mon.noname-c con 0x7f059c1439b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c06f2a0 con 0x7f059c1470b0 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c146b90 con 0x7f059c1439b0 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c0b2ca0 con 0x7f059c0b4330 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:02.240 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.259+0000 7f05a45db640 10 monclient(hunting): authenticate will time out at 1293.527893s 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a2377640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f059c0b4330 0x7f059c0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54786/0 (socket says 172.21.15.107:54786) 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a2377640 1 -- 172.21.15.107:0/4223611419 learned_addr learned my addr 172.21.15.107:0/4223611419 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a1375640 1 -- 172.21.15.107:0/4223611419 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140850251 0 0) 0x7f059c0b2ca0 con 0x7f059c0b4330 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a1375640 10 monclient(hunting): _negotiate 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a1375640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a1375640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a1375640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a1375640 1 -- 172.21.15.107:0/4223611419 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0588003520 con 0x7f059c0b4330 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a1375640 1 -- 172.21.15.107:0/4223611419 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057012376 0 0) 0x7f0588003520 con 0x7f059c0b4330 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a1375640 1 -- 172.21.15.107:0/4223611419 >> v1:172.21.15.129:6789/0 conn(0x7f059c1439b0 legacy=0x7f059c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:06:02.241 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.260+0000 7f05a1375640 1 -- 172.21.15.107:0/4223611419 >> v1:172.21.15.107:6789/0 conn(0x7f059c1470b0 legacy=0x7f059c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:06:02.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: _finish_hunting 0 2024-03-29T11:06:02.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 1 monclient: found mon.noname-b 2024-03-29T11:06:02.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:02.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:06:02.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 1 -- 172.21.15.107:0/4223611419 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c06e0c0 con 0x7f059c0b4330 2024-03-29T11:06:02.244 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: _finish_auth 0 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: _check_auth_tickets 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 1 -- 172.21.15.107:0/4223611419 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f058c001690 con 0x7f059c0b4330 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: dump: 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:02.245 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a45db640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a45db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 1 -- 172.21.15.107:0/4223611419 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f058c001860 con 0x7f059c0b4330 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 1 -- 172.21.15.107:0/4223611419 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f058c001cb0 con 0x7f059c0b4330 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a1375640 10 monclient: dump: 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:02.246 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a45db640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.262+0000 7f05a45db640 4 set_mon_vals no callback set 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.263+0000 7f05a45db640 10 monclient: shutdown 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.263+0000 7f05a45db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.263+0000 7f05a45db640 1 -- 172.21.15.107:0/4223611419 >> v1:172.21.15.107:6790/0 conn(0x7f059c0b4330 legacy=0x7f059c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:02.247 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.263+0000 7f05a45db640 1 -- 172.21.15.107:0/4223611419 shutdown_connections 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.263+0000 7f05a45db640 1 -- 172.21.15.107:0/4223611419 >> 172.21.15.107:0/4223611419 conn(0x7f059c066a00 msgr2=0x7f059c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.264+0000 7f05a45db640 1 -- 172.21.15.107:0/4223611419 shutdown_connections 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.264+0000 7f05a45db640 1 -- 172.21.15.107:0/4223611419 wait complete. 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.264+0000 7f05a45db640 10 monclient: build_initial_monmap 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.264+0000 7f05a45db640 10 monclient: monmap: 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:06:02.248 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:02.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.265+0000 7f05a45db640 1 librados: starting msgr at 2024-03-29T11:06:02.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.265+0000 7f05a45db640 1 librados: starting objecter 2024-03-29T11:06:02.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.265+0000 7f05a45db640 1 Processor -- start 2024-03-29T11:06:02.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.265+0000 7f05a45db640 1 -- start start 2024-03-29T11:06:02.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.265+0000 7f05a45db640 1 librados: setting wanted keys 2024-03-29T11:06:02.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.265+0000 7f05a45db640 1 librados: calling monclient init 2024-03-29T11:06:02.249 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.265+0000 7f05a45db640 10 monclient: init 2024-03-29T11:06:02.252 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.266+0000 7f05a45db640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.266+0000 7f05a45db640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a45db640 10 monclient(hunting): picked mon.noname-c con 0x7f059c1439b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a45db640 10 monclient(hunting): picked mon.noname-a con 0x7f059c1470b0 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a45db640 10 monclient(hunting): picked mon.noname-b con 0x7f059c1dc640 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a45db640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c1dbaa0 con 0x7f059c1470b0 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a45db640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c11f890 con 0x7f059c1dc640 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a45db640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c144350 con 0x7f059c1439b0 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a45db640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a45db640 10 monclient(hunting): authenticate will time out at 1293.535065s 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a2377640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f059c1439b0 0x7f059c1dbf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54922/0 (socket says 172.21.15.107:54922) 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f05a2377640 1 -- 172.21.15.107:0/2157834996 learned_addr learned my addr 172.21.15.107:0/2157834996 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 client.?.objecter ms_handle_connect 0x7f059c1439b0 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:02.253 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245381723 0 0) 0x7f059c144350 con 0x7f059c1439b0 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0580004eb0 con 0x7f059c1439b0 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 141540230 0 0) 0x7f0580004eb0 con 0x7f059c1439b0 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f059c144350 con 0x7f059c1439b0 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f058c001b70 con 0x7f059c1439b0 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 monclient(hunting): dump: 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:02.254 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1973930943 0 0) 0x7f059c144350 con 0x7f059c1439b0 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 >> v1:172.21.15.129:6789/0 conn(0x7f059c1dc640 legacy=0x7f059c1e6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.267+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 >> v1:172.21.15.107:6789/0 conn(0x7f059c1470b0 legacy=0x7f059c1202e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 1 monclient: found mon.c 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:02.255 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c1416c0 con 0x7f059c1439b0 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 10 monclient: _finish_auth 0 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a45db640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a45db640 10 monclient: _renew_subs 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a45db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059c1209f0 con 0x7f059c1439b0 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f058c004bf0 con 0x7f059c1439b0 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a0b74640 4 set_mon_vals no callback set 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f058c004f40 con 0x7f059c1439b0 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a45db640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:02.256 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f0592ffd640 10 monclient: dump: 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a45db640 10 monclient: _renew_subs 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a45db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c120d40 con 0x7f059c1439b0 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.270+0000 7f05a45db640 1 librados: init done 2024-03-29T11:06:02.257 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.272+0000 7f05a45db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.272+0000 7f05a45db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.272+0000 7f05a45db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.272+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059c075f10 con 0x7f059c1439b0 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.272+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f058c01af10 con 0x7f059c1439b0 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.273+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f058c084c70 con 0x7f059c1439b0 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.273+0000 7f0592ffd640 10 client.4601.objecter ms_dispatch 0x7f059c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.273+0000 7f0592ffd640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.273+0000 7f0592ffd640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.273+0000 7f0592ffd640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.273+0000 7f05a1b76640 10 monclient: get_auth_request con 0x7f058006dbd0 auth_method 0 2024-03-29T11:06:02.258 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.273+0000 7f0592ffd640 10 client.4601.objecter ms_handle_connect 0x7f058006dbd0 2024-03-29T11:06:02.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.279+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f058c0505b0 con 0x7f059c1439b0 2024-03-29T11:06:02.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.279+0000 7f0592ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:02.259 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.279+0000 7f0592ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.554+0000 7f05a45db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.554+0000 7f05a45db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.554+0000 7f05a45db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.554+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f059c06f770 con 0x7f059c1439b0 2024-03-29T11:06:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.555+0000 7f0592ffd640 1 -- 172.21.15.107:0/2157834996 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f058c084320 con 0x7f059c1439b0 2024-03-29T11:06:02.534 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.555+0000 7f0592ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:02.535 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.555+0000 7f0592ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:02.535 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:06:02.535 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:02.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.557+0000 7f05a45db640 10 librados: watch_flush enter 2024-03-29T11:06:02.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.557+0000 7f05a45db640 10 librados: watch_flush exit 2024-03-29T11:06:02.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T11:06:02.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 10 client.4601.objecter successfully canceled tick 2024-03-29T11:06:02.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 >> v1:172.21.15.107:6816/882194755 conn(0x7f058006dbd0 legacy=0x7f0580070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:02.537 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 10 monclient: shutdown 2024-03-29T11:06:02.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:02.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 >> v1:172.21.15.107:6790/0 conn(0x7f059c1439b0 legacy=0x7f059c1dbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:02.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 shutdown_connections 2024-03-29T11:06:02.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 >> 172.21.15.107:0/2157834996 conn(0x7f059c066a00 msgr2=0x7f059c0b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:02.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 shutdown_connections 2024-03-29T11:06:02.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 1 -- 172.21.15.107:0/2157834996 wait complete. 2024-03-29T11:06:02.538 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:02.558+0000 7f05a45db640 1 librados: shutdown 2024-03-29T11:06:02.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:09.358 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:09.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.507+0000 7f1fedfaf640 10 monclient: get_monmap_and_config 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.507+0000 7f1fedfaf640 10 monclient: build_initial_monmap 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.507+0000 7f1fedfaf640 10 monclient: monmap: 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:06:09.509652+0000 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:06:09.509652+0000 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:09.488 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:06:09.489 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:06:09.489 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:09.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.508+0000 7f1fedfaf640 1 Processor -- start 2024-03-29T11:06:09.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.508+0000 7f1fedfaf640 1 -- start start 2024-03-29T11:06:09.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.508+0000 7f1fedfaf640 10 monclient: init 2024-03-29T11:06:09.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:09.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:06:09.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe81457f0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe813f650 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe80a9d70 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe80aaef0 con 0x7f1fe80a9d70 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe8145bd0 con 0x7f1fe81457f0 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe8145eb0 con 0x7f1fe813f650 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fedfaf640 10 monclient(hunting): authenticate will time out at 1300.778658s 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fe6ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1fe813f650 0x7f1fe80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:43590/0 (socket says 172.21.15.107:43590) 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.509+0000 7f1fe6ffd640 1 -- 172.21.15.107:0/137578737 learned_addr learned my addr 172.21.15.107:0/137578737 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.510+0000 7f1fe67fc640 1 -- 172.21.15.107:0/137578737 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277771406 0 0) 0x7f1fe8145eb0 con 0x7f1fe813f650 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.510+0000 7f1fe67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.510+0000 7f1fe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:09.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.510+0000 7f1fe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:09.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.510+0000 7f1fe67fc640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:06:09.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.510+0000 7f1fe67fc640 1 -- 172.21.15.107:0/137578737 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd0003520 con 0x7f1fe813f650 2024-03-29T11:06:09.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.510+0000 7f1fe67fc640 1 -- 172.21.15.107:0/137578737 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4150215468 0 0) 0x7f1fd0003520 con 0x7f1fe813f650 2024-03-29T11:06:09.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.510+0000 7f1fe67fc640 1 -- 172.21.15.107:0/137578737 >> v1:172.21.15.129:6789/0 conn(0x7f1fe81457f0 legacy=0x7f1fe813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:09.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.510+0000 7f1fe67fc640 1 -- 172.21.15.107:0/137578737 >> v1:172.21.15.107:6789/0 conn(0x7f1fe80a9d70 legacy=0x7f1fe814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:06:09.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.512+0000 7f1fe67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:06:09.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 1 monclient: found mon.noname-b 2024-03-29T11:06:09.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:09.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:06:09.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 1 -- 172.21.15.107:0/137578737 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe8146340 con 0x7f1fe813f650 2024-03-29T11:06:09.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: _finish_auth 0 2024-03-29T11:06:09.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 1 -- 172.21.15.107:0/137578737 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1fdc001690 con 0x7f1fe813f650 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: dump: 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.496 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fedfaf640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fedfaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 1 -- 172.21.15.107:0/137578737 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fdc001860 con 0x7f1fe813f650 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 1 -- 172.21.15.107:0/137578737 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1fdc001cb0 con 0x7f1fe813f650 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fe67fc640 10 monclient: dump: 2024-03-29T11:06:09.497 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fedfaf640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.513+0000 7f1fedfaf640 4 set_mon_vals no callback set 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.515+0000 7f1fedfaf640 10 monclient: shutdown 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.515+0000 7f1fedfaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:09.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.515+0000 7f1fedfaf640 1 -- 172.21.15.107:0/137578737 >> v1:172.21.15.107:6790/0 conn(0x7f1fe813f650 legacy=0x7f1fe80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:09.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.515+0000 7f1fedfaf640 1 -- 172.21.15.107:0/137578737 shutdown_connections 2024-03-29T11:06:09.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.515+0000 7f1fedfaf640 1 -- 172.21.15.107:0/137578737 >> 172.21.15.107:0/137578737 conn(0x7f1fe8066a00 msgr2=0x7f1fe8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:09.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.516+0000 7f1fedfaf640 1 -- 172.21.15.107:0/137578737 shutdown_connections 2024-03-29T11:06:09.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.516+0000 7f1fedfaf640 1 -- 172.21.15.107:0/137578737 wait complete. 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.517+0000 7f1fedfaf640 10 monclient: build_initial_monmap 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.517+0000 7f1fedfaf640 10 monclient: monmap: 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:09.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.517+0000 7f1fedfaf640 1 librados: starting msgr at 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.517+0000 7f1fedfaf640 1 librados: starting objecter 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.517+0000 7f1fedfaf640 1 Processor -- start 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.518+0000 7f1fedfaf640 1 -- start start 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.518+0000 7f1fedfaf640 1 librados: setting wanted keys 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.518+0000 7f1fedfaf640 1 librados: calling monclient init 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.518+0000 7f1fedfaf640 10 monclient: init 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.519+0000 7f1fedfaf640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.519+0000 7f1fedfaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.519+0000 7f1fedfaf640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe80a9d70 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.519+0000 7f1fedfaf640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe813f650 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.519+0000 7f1fe6ffd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1fe813f650 0x7f1fe8136b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44066/0 (socket says 172.21.15.107:44066) 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.519+0000 7f1fe6ffd640 1 -- 172.21.15.107:0/2881960680 learned_addr learned my addr 172.21.15.107:0/2881960680 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.519+0000 7f1fcffff640 10 client.?.objecter ms_handle_connect 0x7f1fe813f650 2024-03-29T11:06:09.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fcffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fcffff640 10 client.?.objecter ms_handle_connect 0x7f1fe80a9d70 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fcffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fedfaf640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe81457f0 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe81429b0 con 0x7f1fe81457f0 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe8137990 con 0x7f1fe80a9d70 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe8142e40 con 0x7f1fe813f650 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fedfaf640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fedfaf640 10 monclient(hunting): authenticate will time out at 1300.789693s 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186706501 0 0) 0x7f1fe8137990 con 0x7f1fe80a9d70 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fcffff640 10 monclient(hunting): _negotiate 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.520+0000 7f1fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:06:09.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd4004eb0 con 0x7f1fe80a9d70 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989130310 0 0) 0x7f1fe8142e40 con 0x7f1fe813f650 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): _negotiate 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe8137990 con 0x7f1fe813f650 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3101496231 0 0) 0x7f1fd4004eb0 con 0x7f1fe80a9d70 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fe8142e40 con 0x7f1fe80a9d70 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1fd8001890 con 0x7f1fe80a9d70 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): dump: 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:09.509 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2467095626 0 0) 0x7f1fe8137990 con 0x7f1fe813f650 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.521+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fd4004eb0 con 0x7f1fe813f650 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1fdc001b70 con 0x7f1fe813f650 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:09.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 10 monclient(hunting): dump: 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 10 client.?.objecter ms_handle_connect 0x7f1fe81457f0 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:09.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2910268645 0 0) 0x7f1fe8142e40 con 0x7f1fe80a9d70 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 >> v1:172.21.15.107:6790/0 conn(0x7f1fe813f650 legacy=0x7f1fe8136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 >> v1:172.21.15.107:6789/0 conn(0x7f1fe81457f0 legacy=0x7f1fe8137280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 10 monclient: _finish_hunting 0 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 1 monclient: found mon.b 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe80ac710 con 0x7f1fe80a9d70 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 10 monclient: _finish_auth 0 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 10 monclient: _check_auth_tickets 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fedfaf640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fedfaf640 10 monclient: _renew_subs 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fedfaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:06:09.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe80ac9f0 con 0x7f1fe80a9d70 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fedfaf640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fedfaf640 10 monclient: _renew_subs 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fedfaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe81d7dd0 con 0x7f1fe80a9d70 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.522+0000 7f1fedfaf640 1 librados: init done 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd80035f0 con 0x7f1fe80a9d70 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fcffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1fd8001d80 con 0x7f1fe80a9d70 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fcffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fcffff640 10 monclient: dump: 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.513 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fedfaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fedfaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fedfaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe8141e60 con 0x7f1fe80a9d70 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.523+0000 7f1fe5ffb640 4 set_mon_vals no callback set 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.524+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f1fd8003980 con 0x7f1fe80a9d70 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.525+0000 7f1fe6ffd640 10 monclient: get_auth_request con 0x7f1fd406de40 auth_method 0 2024-03-29T11:06:09.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.525+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f1fd8083720 con 0x7f1fe80a9d70 2024-03-29T11:06:09.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.525+0000 7f1fcffff640 10 client.4486.objecter ms_dispatch 0x7f1fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:09.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.525+0000 7f1fcffff640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:09.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.525+0000 7f1fcffff640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:09.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.525+0000 7f1fcffff640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-29T11:06:09.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.525+0000 7f1fcffff640 10 client.4486.objecter ms_handle_connect 0x7f1fd406de40 2024-03-29T11:06:09.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.532+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fd804f000 con 0x7f1fe80a9d70 2024-03-29T11:06:09.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.532+0000 7f1fcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:09.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.532+0000 7f1fcffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:09.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.843+0000 7f1fedfaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:09.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.843+0000 7f1fedfaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:09.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.843+0000 7f1fedfaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:06:09.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.843+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe8075f10 con 0x7f1fe80a9d70 2024-03-29T11:06:09.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.845+0000 7f1fcffff640 1 -- 172.21.15.107:0/2881960680 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f1fd8050f30 con 0x7f1fe80a9d70 2024-03-29T11:06:09.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.845+0000 7f1fcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:09.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.845+0000 7f1fcffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:09.827 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:06:09.827 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:09.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.849+0000 7f1fedfaf640 10 librados: watch_flush enter 2024-03-29T11:06:09.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.849+0000 7f1fedfaf640 10 librados: watch_flush exit 2024-03-29T11:06:09.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.849+0000 7f1fedfaf640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-29T11:06:09.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.849+0000 7f1fedfaf640 10 client.4486.objecter successfully canceled tick 2024-03-29T11:06:09.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.850+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 >> v1:172.21.15.107:6816/882194755 conn(0x7f1fd406de40 legacy=0x7f1fd4070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:09.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.850+0000 7f1fedfaf640 10 monclient: shutdown 2024-03-29T11:06:09.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.850+0000 7f1fedfaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:09.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.850+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 >> v1:172.21.15.129:6789/0 conn(0x7f1fe80a9d70 legacy=0x7f1fe81d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:09.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.850+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 shutdown_connections 2024-03-29T11:06:09.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.850+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 >> 172.21.15.107:0/2881960680 conn(0x7f1fe8066a00 msgr2=0x7f1fe80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:09.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.850+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 shutdown_connections 2024-03-29T11:06:09.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.850+0000 7f1fedfaf640 1 -- 172.21.15.107:0/2881960680 wait complete. 2024-03-29T11:06:09.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:09.850+0000 7f1fedfaf640 1 librados: shutdown 2024-03-29T11:06:09.844 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:16.650 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:16.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.815+0000 7fb458425640 10 monclient: get_monmap_and_config 2024-03-29T11:06:16.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.815+0000 7fb458425640 10 monclient: build_initial_monmap 2024-03-29T11:06:16.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.815+0000 7fb458425640 10 monclient: monmap: 2024-03-29T11:06:16.794 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:16.794 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:16.795 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:06:16.816383+0000 2024-03-29T11:06:16.795 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:06:16.816383+0000 2024-03-29T11:06:16.795 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:16.795 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:16.795 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:16.795 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:06:16.795 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:06:16.795 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:16.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.816+0000 7fb458425640 1 Processor -- start 2024-03-29T11:06:16.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.816+0000 7fb458425640 1 -- start start 2024-03-29T11:06:16.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.816+0000 7fb458425640 10 monclient: init 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.817+0000 7fb458425640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.817+0000 7fb458425640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.818+0000 7fb458425640 10 monclient(hunting): picked mon.noname-c con 0x7fb4501422b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.818+0000 7fb458425640 10 monclient(hunting): picked mon.noname-a con 0x7fb450142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.818+0000 7fb458425640 10 monclient(hunting): picked mon.noname-b con 0x7fb4500b17d0 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.818+0000 7fb458425640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb450142c80 con 0x7fb450142690 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.818+0000 7fb458425640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4500aae40 con 0x7fb4501422b0 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.818+0000 7fb458425640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45013fbd0 con 0x7fb4500b17d0 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.818+0000 7fb458425640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:16.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.818+0000 7fb458425640 10 monclient(hunting): authenticate will time out at 1308.086076s 2024-03-29T11:06:16.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.820+0000 7fb4569c2640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb4500b17d0 0x7fb4500b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33014/0 (socket says 172.21.15.107:33014) 2024-03-29T11:06:16.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.821+0000 7fb4569c2640 1 -- 172.21.15.107:0/1161419288 learned_addr learned my addr 172.21.15.107:0/1161419288 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:16.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173019321 0 0) 0x7fb4500aae40 con 0x7fb4501422b0 2024-03-29T11:06:16.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): _negotiate 2024-03-29T11:06:16.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:16.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:16.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:06:16.808 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb43c003520 con 0x7fb4501422b0 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105986014 0 0) 0x7fb450142c80 con 0x7fb450142690 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): _negotiate 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4500aae40 con 0x7fb450142690 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915180794 0 0) 0x7fb45013fbd0 con 0x7fb4500b17d0 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): _negotiate 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb450142c80 con 0x7fb4500b17d0 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4205258231 0 0) 0x7fb43c003520 con 0x7fb4501422b0 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.825+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 >> v1:172.21.15.107:6790/0 conn(0x7fb4500b17d0 legacy=0x7fb4500b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 >> v1:172.21.15.107:6789/0 conn(0x7fb450142690 legacy=0x7fb4500b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:16.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: _finish_hunting 0 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 1 monclient: found mon.noname-c 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb450151940 con 0x7fb4501422b0 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: _finish_auth 0 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: _check_auth_tickets 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb448001a10 con 0x7fb4501422b0 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: dump: 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:16.810 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb448001bf0 con 0x7fb4501422b0 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 1 -- 172.21.15.107:0/1161419288 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb448001f80 con 0x7fb4501422b0 2024-03-29T11:06:16.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:16.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:06:16.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.826+0000 7fb4551bf640 10 monclient: dump: 2024-03-29T11:06:16.812 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:16.812 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:16.812 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:16.812 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:16.812 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.827+0000 7fb458425640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.827+0000 7fb458425640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.827+0000 7fb458425640 4 set_mon_vals no callback set 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.828+0000 7fb458425640 10 monclient: shutdown 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.828+0000 7fb458425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.828+0000 7fb458425640 1 -- 172.21.15.107:0/1161419288 >> v1:172.21.15.129:6789/0 conn(0x7fb4501422b0 legacy=0x7fb45013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.829+0000 7fb458425640 1 -- 172.21.15.107:0/1161419288 shutdown_connections 2024-03-29T11:06:16.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.829+0000 7fb458425640 1 -- 172.21.15.107:0/1161419288 >> 172.21.15.107:0/1161419288 conn(0x7fb450066a00 msgr2=0x7fb450066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:16.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.830+0000 7fb458425640 1 -- 172.21.15.107:0/1161419288 shutdown_connections 2024-03-29T11:06:16.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.830+0000 7fb458425640 1 -- 172.21.15.107:0/1161419288 wait complete. 2024-03-29T11:06:16.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.831+0000 7fb458425640 10 monclient: build_initial_monmap 2024-03-29T11:06:16.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.831+0000 7fb458425640 10 monclient: monmap: 2024-03-29T11:06:16.816 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:16.816 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:16.816 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 1 librados: starting msgr at 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 1 librados: starting objecter 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 1 Processor -- start 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 1 -- start start 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 1 librados: setting wanted keys 2024-03-29T11:06:16.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 1 librados: calling monclient init 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 10 monclient: init 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 10 monclient(hunting): picked mon.noname-b con 0x7fb4500b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 10 monclient(hunting): picked mon.noname-c con 0x7fb4501422b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 10 monclient(hunting): picked mon.noname-a con 0x7fb450142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4501328c0 con 0x7fb450142690 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.832+0000 7fb458425640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45006e470 con 0x7fb4500b17d0 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.833+0000 7fb458425640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45006e600 con 0x7fb4501422b0 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.833+0000 7fb458425640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.833+0000 7fb458425640 10 monclient(hunting): authenticate will time out at 1308.100990s 2024-03-29T11:06:16.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.836+0000 7fb4559c0640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fb4501422b0 0x7fb4501dbdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33284/0 (socket says 172.21.15.107:33284) 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.836+0000 7fb4559c0640 1 -- 172.21.15.107:0/1576879169 learned_addr learned my addr 172.21.15.107:0/1576879169 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.836+0000 7fb446ffd640 10 client.?.objecter ms_handle_connect 0x7fb4501422b0 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.836+0000 7fb446ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.837+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642999372 0 0) 0x7fb45006e600 con 0x7fb4501422b0 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.837+0000 7fb446ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.837+0000 7fb446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.837+0000 7fb446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.837+0000 7fb446ffd640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.837+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb434004eb0 con 0x7fb4501422b0 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.838+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3436865029 0 0) 0x7fb434004eb0 con 0x7fb4501422b0 2024-03-29T11:06:16.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.838+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb45006e600 con 0x7fb4501422b0 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.838+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb440001930 con 0x7fb4501422b0 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.838+0000 7fb446ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.838+0000 7fb446ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.838+0000 7fb446ffd640 10 monclient(hunting): dump: 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:16.820 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.838+0000 7fb446ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 client.?.objecter ms_handle_connect 0x7fb4500b17d0 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 client.?.objecter ms_handle_connect 0x7fb450142690 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:16.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518102854 0 0) 0x7fb45006e470 con 0x7fb4500b17d0 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb434004eb0 con 0x7fb4500b17d0 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672382868 0 0) 0x7fb4501328c0 con 0x7fb450142690 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45006e470 con 0x7fb450142690 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 914272205 0 0) 0x7fb45006e600 con 0x7fb4501422b0 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 >> v1:172.21.15.129:6789/0 conn(0x7fb4500b17d0 legacy=0x7fb450131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:16.822 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 >> v1:172.21.15.107:6789/0 conn(0x7fb450142690 legacy=0x7fb4501dc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:16.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:06:16.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 1 monclient: found mon.c 2024-03-29T11:06:16.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:16.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:16.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb450071d70 con 0x7fb4501422b0 2024-03-29T11:06:16.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient: _finish_auth 0 2024-03-29T11:06:16.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:06:16.823 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.842+0000 7fb446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:16.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.844+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb440003610 con 0x7fb4501422b0 2024-03-29T11:06:16.824 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.844+0000 7fb446ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.844+0000 7fb458425640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.844+0000 7fb458425640 10 monclient: _renew_subs 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.844+0000 7fb458425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.844+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb450071fc0 con 0x7fb4501422b0 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.844+0000 7fb447fff640 4 set_mon_vals no callback set 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.844+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fb440003a20 con 0x7fb4501422b0 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.845+0000 7fb446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.845+0000 7fb446ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.845+0000 7fb446ffd640 10 monclient: dump: 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:16.825 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.845+0000 7fb458425640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.845+0000 7fb458425640 10 monclient: _renew_subs 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.845+0000 7fb458425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.845+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4501e77e0 con 0x7fb4501422b0 2024-03-29T11:06:16.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.845+0000 7fb458425640 1 librados: init done 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.847+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fb440003c00 con 0x7fb4501422b0 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.847+0000 7fb458425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.847+0000 7fb458425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.847+0000 7fb458425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.847+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4501326c0 con 0x7fb4501422b0 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.847+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fb440084b20 con 0x7fb4501422b0 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.847+0000 7fb446ffd640 10 client.4613.objecter ms_dispatch 0x7fb450000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.847+0000 7fb446ffd640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.847+0000 7fb446ffd640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.848+0000 7fb446ffd640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:06:16.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.849+0000 7fb4561c1640 10 monclient: get_auth_request con 0x7fb43406ddb0 auth_method 0 2024-03-29T11:06:16.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.849+0000 7fb446ffd640 10 client.4613.objecter ms_handle_connect 0x7fb43406ddb0 2024-03-29T11:06:16.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.856+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb440050400 con 0x7fb4501422b0 2024-03-29T11:06:16.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.856+0000 7fb446ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:16.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:16.856+0000 7fb446ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:17.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.183+0000 7fb458425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:17.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.183+0000 7fb458425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:17.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.183+0000 7fb458425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:17.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.183+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb450075f10 con 0x7fb4501422b0 2024-03-29T11:06:17.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.184+0000 7fb446ffd640 1 -- 172.21.15.107:0/1576879169 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fb440052330 con 0x7fb4501422b0 2024-03-29T11:06:17.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.184+0000 7fb446ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:17.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.184+0000 7fb446ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:17.166 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:06:17.166 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:17.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.187+0000 7fb458425640 10 librados: watch_flush enter 2024-03-29T11:06:17.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.190+0000 7fb458425640 10 librados: watch_flush exit 2024-03-29T11:06:17.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.190+0000 7fb458425640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:06:17.169 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.190+0000 7fb458425640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:06:17.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.190+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 >> v1:172.21.15.107:6816/882194755 conn(0x7fb43406ddb0 legacy=0x7fb434070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:17.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.190+0000 7fb458425640 10 monclient: shutdown 2024-03-29T11:06:17.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.190+0000 7fb458425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:17.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.190+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 >> v1:172.21.15.107:6790/0 conn(0x7fb4501422b0 legacy=0x7fb4501dbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:17.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.191+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 shutdown_connections 2024-03-29T11:06:17.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.191+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 >> 172.21.15.107:0/1576879169 conn(0x7fb450066a00 msgr2=0x7fb45013c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:17.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.192+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 shutdown_connections 2024-03-29T11:06:17.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.192+0000 7fb458425640 1 -- 172.21.15.107:0/1576879169 wait complete. 2024-03-29T11:06:17.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:17.194+0000 7fb458425640 1 librados: shutdown 2024-03-29T11:06:17.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:24.001 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:24.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.173+0000 7ff5aa8e0640 10 monclient: get_monmap_and_config 2024-03-29T11:06:24.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.173+0000 7ff5aa8e0640 10 monclient: build_initial_monmap 2024-03-29T11:06:24.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.173+0000 7ff5aa8e0640 10 monclient: monmap: 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:06:24.174633+0000 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:06:24.174633+0000 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:06:24.154 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:24.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.174+0000 7ff5aa8e0640 1 Processor -- start 2024-03-29T11:06:24.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.175+0000 7ff5aa8e0640 1 -- start start 2024-03-29T11:06:24.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.175+0000 7ff5aa8e0640 10 monclient: init 2024-03-29T11:06:24.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.175+0000 7ff5aa8e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:24.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.175+0000 7ff5aa8e0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:06:24.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.175+0000 7ff5aa8e0640 10 monclient(hunting): picked mon.noname-c con 0x7ff59c034850 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:24.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.175+0000 7ff5aa8e0640 10 monclient(hunting): picked mon.noname-b con 0x7ff59c0c9550 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:24.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.178+0000 7ff5aa8e0640 10 monclient(hunting): picked mon.noname-a con 0x7ff59c02aac0 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:24.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.178+0000 7ff5a8e7d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff59c02aac0 0x7ff59c0cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49143/0 (socket says 172.21.15.107:49143) 2024-03-29T11:06:24.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.178+0000 7ff5a8e7d640 1 -- 172.21.15.107:0/3965696561 learned_addr learned my addr 172.21.15.107:0/3965696561 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:24.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.178+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3965696561 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c02c440 con 0x7ff59c02aac0 2024-03-29T11:06:24.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.178+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3965696561 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c02aed0 con 0x7ff59c034850 2024-03-29T11:06:24.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.178+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3965696561 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c02b190 con 0x7ff59c0c9550 2024-03-29T11:06:24.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.178+0000 7ff5aa8e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:24.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.178+0000 7ff5aa8e0640 10 monclient(hunting): authenticate will time out at 1315.446662s 2024-03-29T11:06:24.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.179+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198099393 0 0) 0x7ff59c02c440 con 0x7ff59c02aac0 2024-03-29T11:06:24.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.179+0000 7ff5a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:24.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.179+0000 7ff5a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:24.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.179+0000 7ff5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:24.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.179+0000 7ff5a2ffd640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:06:24.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.179+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff594003520 con 0x7ff59c02aac0 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.181+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962961673 0 0) 0x7ff59c02aed0 con 0x7ff59c034850 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.181+0000 7ff5a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.181+0000 7ff5a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.181+0000 7ff5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.181+0000 7ff5a2ffd640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.181+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c02c440 con 0x7ff59c034850 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405380965 0 0) 0x7ff59c02b190 con 0x7ff59c0c9550 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:06:24.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c02aed0 con 0x7ff59c0c9550 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2363040651 0 0) 0x7ff59c02c440 con 0x7ff59c034850 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 >> v1:172.21.15.107:6790/0 conn(0x7ff59c0c9550 legacy=0x7ff59c0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 >> v1:172.21.15.107:6789/0 conn(0x7ff59c02aac0 legacy=0x7ff59c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c033060 con 0x7ff59c034850 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff598001690 con 0x7ff59c034850 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:24.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: dump: 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5aa8e0640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5aa8e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:24.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5980017f0 con 0x7ff59c034850 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5aa8e0640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5aa8e0640 4 set_mon_vals no callback set 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 1 -- 172.21.15.107:0/3965696561 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff598001c00 con 0x7ff59c034850 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.182+0000 7ff5a2ffd640 10 monclient: dump: 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:24.166 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:24.167 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:24.167 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:24.167 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:24.167 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:24.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.183+0000 7ff5aa8e0640 10 monclient: shutdown 2024-03-29T11:06:24.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.183+0000 7ff5aa8e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.183+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3965696561 >> v1:172.21.15.129:6789/0 conn(0x7ff59c034850 legacy=0x7ff59c0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.183+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3965696561 shutdown_connections 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.183+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3965696561 >> 172.21.15.107:0/3965696561 conn(0x7ff59c0058f0 msgr2=0x7ff59c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.186+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3965696561 shutdown_connections 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.187+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3965696561 wait complete. 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.188+0000 7ff5aa8e0640 10 monclient: build_initial_monmap 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.188+0000 7ff5aa8e0640 10 monclient: monmap: 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:24.172 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.188+0000 7ff5aa8e0640 1 librados: starting msgr at 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.188+0000 7ff5aa8e0640 1 librados: starting objecter 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.188+0000 7ff5aa8e0640 1 Processor -- start 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.189+0000 7ff5aa8e0640 1 -- start start 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.189+0000 7ff5aa8e0640 1 librados: setting wanted keys 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.189+0000 7ff5aa8e0640 1 librados: calling monclient init 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.189+0000 7ff5aa8e0640 10 monclient: init 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.189+0000 7ff5aa8e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.189+0000 7ff5aa8e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:06:24.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.189+0000 7ff5aa8e0640 10 monclient(hunting): picked mon.noname-a con 0x7ff59c02aac0 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.189+0000 7ff5aa8e0640 10 monclient(hunting): picked mon.noname-c con 0x7ff59c0c9550 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.190+0000 7ff5aa8e0640 10 monclient(hunting): picked mon.noname-b con 0x7ff59c0cb3c0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.190+0000 7ff5aa8e0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0bbd20 con 0x7ff59c02aac0 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.190+0000 7ff5aa8e0640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0be5a0 con 0x7ff59c0cb3c0 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.190+0000 7ff5aa8e0640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0bd790 con 0x7ff59c0c9550 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.190+0000 7ff5aa8e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.190+0000 7ff5aa8e0640 10 monclient(hunting): authenticate will time out at 1315.458425s 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.190+0000 7ff5a37fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7ff59c0c9550 0x7ff59c0bde90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:59406/0 (socket says 172.21.15.107:59406) 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.190+0000 7ff5a37fe640 1 -- 172.21.15.107:0/3607168991 learned_addr learned my addr 172.21.15.107:0/3607168991 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 10 client.?.objecter ms_handle_connect 0x7ff59c0c9550 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215355968 0 0) 0x7ff59c0bd790 con 0x7ff59c0c9550 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:06:24.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff588004eb0 con 0x7ff59c0c9550 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357301995 0 0) 0x7ff588004eb0 con 0x7ff59c0c9550 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.191+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff59c0bd790 con 0x7ff59c0c9550 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.192+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff590001910 con 0x7ff59c0c9550 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.192+0000 7ff5a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.192+0000 7ff5a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.192+0000 7ff5a0ff9640 10 monclient(hunting): dump: 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:24.175 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:24.176 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:24.176 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:24.176 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:24.176 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:24.176 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:24.176 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.192+0000 7ff5a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:24.176 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.192+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1712212116 0 0) 0x7ff59c0bd790 con 0x7ff59c0c9550 2024-03-29T11:06:24.176 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.192+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 >> v1:172.21.15.129:6789/0 conn(0x7ff59c0cb3c0 legacy=0x7ff59c16b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:24.176 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.192+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 >> v1:172.21.15.107:6789/0 conn(0x7ff59c02aac0 legacy=0x7ff59c0bc4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:06:24.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.195+0000 7ff5a0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:06:24.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.195+0000 7ff5a0ff9640 1 monclient: found mon.c 2024-03-29T11:06:24.177 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.195+0000 7ff5a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.195+0000 7ff5a0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.195+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c0bda70 con 0x7ff59c0c9550 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.195+0000 7ff5a0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5a0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5aa8e0640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5aa8e0640 10 monclient: _renew_subs 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5aa8e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c0bbf90 con 0x7ff59c0c9550 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5aa8e0640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5aa8e0640 10 monclient: _renew_subs 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5aa8e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c0bb5e0 con 0x7ff59c0c9550 2024-03-29T11:06:24.178 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.196+0000 7ff5aa8e0640 1 librados: init done 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.197+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff590001d40 con 0x7ff59c0c9550 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.197+0000 7ff5a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.197+0000 7ff5a27fc640 4 set_mon_vals no callback set 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.198+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7ff590003a60 con 0x7ff59c0c9550 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.198+0000 7ff5a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.198+0000 7ff5a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.198+0000 7ff5a0ff9640 10 monclient: dump: 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:24.179 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:24.180 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:24.180 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:24.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.199+0000 7ff5aa8e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:24.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.199+0000 7ff5aa8e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:24.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.199+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7ff590003bc0 con 0x7ff59c0c9550 2024-03-29T11:06:24.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.199+0000 7ff5aa8e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:24.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.199+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff584005080 con 0x7ff59c0c9550 2024-03-29T11:06:24.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.202+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7ff590084b10 con 0x7ff59c0c9550 2024-03-29T11:06:24.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.202+0000 7ff5a0ff9640 10 client.4619.objecter ms_dispatch 0x7ff5a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:24.181 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.202+0000 7ff5a0ff9640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:24.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.202+0000 7ff5a0ff9640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:24.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.202+0000 7ff5a0ff9640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T11:06:24.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.202+0000 7ff5a3fff640 10 monclient: get_auth_request con 0x7ff58806dc20 auth_method 0 2024-03-29T11:06:24.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.202+0000 7ff5a0ff9640 10 client.4619.objecter ms_handle_connect 0x7ff58806dc20 2024-03-29T11:06:24.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.211+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5900503f0 con 0x7ff59c0c9550 2024-03-29T11:06:24.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.211+0000 7ff5a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:24.190 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.211+0000 7ff5a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:24.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.530+0000 7ff5aa8e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:24.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.530+0000 7ff5aa8e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:24.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.530+0000 7ff5aa8e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:24.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.530+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff584005330 con 0x7ff59c0c9550 2024-03-29T11:06:24.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.531+0000 7ff5a0ff9640 1 -- 172.21.15.107:0/3607168991 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7ff590052320 con 0x7ff59c0c9550 2024-03-29T11:06:24.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.531+0000 7ff5a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:24.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.531+0000 7ff5a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:24.511 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:06:24.511 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:24.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.535+0000 7ff5aa8e0640 10 librados: watch_flush enter 2024-03-29T11:06:24.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.535+0000 7ff5aa8e0640 10 librados: watch_flush exit 2024-03-29T11:06:24.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.535+0000 7ff5aa8e0640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T11:06:24.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.535+0000 7ff5aa8e0640 10 client.4619.objecter successfully canceled tick 2024-03-29T11:06:24.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.538+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 >> v1:172.21.15.107:6816/882194755 conn(0x7ff58806dc20 legacy=0x7ff5880700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:24.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.538+0000 7ff5aa8e0640 10 monclient: shutdown 2024-03-29T11:06:24.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.538+0000 7ff5aa8e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:24.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.538+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 >> v1:172.21.15.107:6790/0 conn(0x7ff59c0c9550 legacy=0x7ff59c0bde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:24.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.538+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 shutdown_connections 2024-03-29T11:06:24.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.539+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 >> 172.21.15.107:0/3607168991 conn(0x7ff59c0058f0 msgr2=0x7ff59c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:24.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.539+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 shutdown_connections 2024-03-29T11:06:24.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.539+0000 7ff5aa8e0640 1 -- 172.21.15.107:0/3607168991 wait complete. 2024-03-29T11:06:24.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:24.539+0000 7ff5aa8e0640 1 librados: shutdown 2024-03-29T11:06:24.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:31.343 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:31.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.504+0000 7fafb8b02640 10 monclient: get_monmap_and_config 2024-03-29T11:06:31.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.504+0000 7fafb8b02640 10 monclient: build_initial_monmap 2024-03-29T11:06:31.483 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.504+0000 7fafb8b02640 10 monclient: monmap: 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:06:31.505636+0000 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:06:31.505636+0000 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:06:31.484 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:31.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.506+0000 7fafb8b02640 1 Processor -- start 2024-03-29T11:06:31.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.506+0000 7fafb8b02640 1 -- start start 2024-03-29T11:06:31.485 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.506+0000 7fafb8b02640 10 monclient: init 2024-03-29T11:06:31.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.507+0000 7fafb8b02640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:31.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.507+0000 7fafb8b02640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:06:31.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.507+0000 7fafb8b02640 10 monclient(hunting): picked mon.noname-c con 0x7fafb41422b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:31.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.507+0000 7fafb8b02640 10 monclient(hunting): picked mon.noname-b con 0x7fafb4142690 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:31.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.507+0000 7fafb1d9b640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fafb4142690 0x7fafb40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:58678/0 (socket says 172.21.15.107:58678) 2024-03-29T11:06:31.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.507+0000 7fafb1d9b640 1 -- 172.21.15.107:0/2711735121 learned_addr learned my addr 172.21.15.107:0/2711735121 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:31.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.508+0000 7fafb8b02640 10 monclient(hunting): picked mon.noname-a con 0x7fafb40b17d0 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:31.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.509+0000 7fafb8b02640 1 -- 172.21.15.107:0/2711735121 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb4142c80 con 0x7fafb40b17d0 2024-03-29T11:06:31.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.509+0000 7fafb8b02640 1 -- 172.21.15.107:0/2711735121 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb40aae40 con 0x7fafb41422b0 2024-03-29T11:06:31.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.509+0000 7fafb8b02640 1 -- 172.21.15.107:0/2711735121 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb413fbd0 con 0x7fafb4142690 2024-03-29T11:06:31.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.509+0000 7fafb8b02640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:31.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.509+0000 7fafb8b02640 10 monclient(hunting): authenticate will time out at 1322.777138s 2024-03-29T11:06:31.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932845161 0 0) 0x7fafb413fbd0 con 0x7fafb4142690 2024-03-29T11:06:31.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 10 monclient(hunting): _negotiate 2024-03-29T11:06:31.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:31.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:31.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9c003520 con 0x7fafb4142690 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733453902 0 0) 0x7fafb40aae40 con 0x7fafb41422b0 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 10 monclient(hunting): _negotiate 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9c003840 con 0x7fafb41422b0 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.511+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1600517043 0 0) 0x7faf9c003520 con 0x7fafb4142690 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 >> v1:172.21.15.129:6789/0 conn(0x7fafb41422b0 legacy=0x7fafb413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 >> v1:172.21.15.107:6789/0 conn(0x7fafb40b17d0 legacy=0x7fafb40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:31.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 10 monclient: _finish_hunting 0 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 1 monclient: found mon.noname-b 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb4151940 con 0x7fafb4142690 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 10 monclient: _finish_auth 0 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 10 monclient: _check_auth_tickets 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb8b02640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.512+0000 7fafb8b02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fafa4001670 con 0x7fafb4142690 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:31.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 10 monclient: dump: 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafa4001850 con 0x7fafb4142690 2024-03-29T11:06:31.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 1 -- 172.21.15.107:0/2711735121 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fafa4001be0 con 0x7fafb4142690 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.515+0000 7fafb159a640 10 monclient: dump: 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:31.498 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.516+0000 7fafb8b02640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.516+0000 7fafb8b02640 4 set_mon_vals no callback set 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.516+0000 7fafb8b02640 10 monclient: shutdown 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.517+0000 7fafb8b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.517+0000 7fafb8b02640 1 -- 172.21.15.107:0/2711735121 >> v1:172.21.15.107:6790/0 conn(0x7fafb4142690 legacy=0x7fafb40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.517+0000 7fafb8b02640 1 -- 172.21.15.107:0/2711735121 shutdown_connections 2024-03-29T11:06:31.499 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.517+0000 7fafb8b02640 1 -- 172.21.15.107:0/2711735121 >> 172.21.15.107:0/2711735121 conn(0x7fafb4066a00 msgr2=0x7fafb4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:31.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.517+0000 7fafb8b02640 1 -- 172.21.15.107:0/2711735121 shutdown_connections 2024-03-29T11:06:31.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.517+0000 7fafb8b02640 1 -- 172.21.15.107:0/2711735121 wait complete. 2024-03-29T11:06:31.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.518+0000 7fafb8b02640 10 monclient: build_initial_monmap 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.519+0000 7fafb8b02640 10 monclient: monmap: 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:06:31.501 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:31.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.519+0000 7fafb8b02640 1 librados: starting msgr at 2024-03-29T11:06:31.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.519+0000 7fafb8b02640 1 librados: starting objecter 2024-03-29T11:06:31.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.519+0000 7fafb8b02640 1 Processor -- start 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.521+0000 7fafb8b02640 1 -- start start 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.521+0000 7fafb8b02640 1 librados: setting wanted keys 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.521+0000 7fafb8b02640 1 librados: calling monclient init 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.521+0000 7fafb8b02640 10 monclient: init 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.522+0000 7fafb8b02640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.522+0000 7fafb8b02640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.522+0000 7fafb8b02640 10 monclient(hunting): picked mon.noname-c con 0x7fafb40b17d0 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.522+0000 7fafb259c640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fafb40b17d0 0x7fafb413bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:58898/0 (socket says 172.21.15.107:58898) 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.522+0000 7fafb259c640 1 -- 172.21.15.107:0/1168180535 learned_addr learned my addr 172.21.15.107:0/1168180535 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:31.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.523+0000 7fafa2ffd640 10 client.?.objecter ms_handle_connect 0x7fafb40b17d0 2024-03-29T11:06:31.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.523+0000 7fafa2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:31.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.523+0000 7fafb8b02640 10 monclient(hunting): picked mon.noname-b con 0x7fafb41422b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:31.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.526+0000 7fafb8b02640 10 monclient(hunting): picked mon.noname-a con 0x7fafb4142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:31.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.526+0000 7fafa2ffd640 10 client.?.objecter ms_handle_connect 0x7fafb4142690 2024-03-29T11:06:31.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.526+0000 7fafa2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:31.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.526+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb41dfa60 con 0x7fafb4142690 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.526+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb406feb0 con 0x7fafb41422b0 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb4070190 con 0x7fafb40b17d0 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 client.?.objecter ms_handle_connect 0x7fafb41422b0 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafb8b02640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafb8b02640 10 monclient(hunting): authenticate will time out at 1322.795279s 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262347581 0 0) 0x7fafb41dfa60 con 0x7fafb4142690 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:31.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf94004eb0 con 0x7fafb4142690 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995706270 0 0) 0x7fafb4070190 con 0x7fafb40b17d0 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.527+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb41dfa60 con 0x7fafb40b17d0 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089658594 0 0) 0x7fafb406feb0 con 0x7fafb41422b0 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:31.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb4070190 con 0x7fafb41422b0 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1456732695 0 0) 0x7faf94004eb0 con 0x7fafb4142690 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafb406feb0 con 0x7fafb4142690 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1340873431 0 0) 0x7fafb41dfa60 con 0x7fafb40b17d0 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf94004eb0 con 0x7fafb40b17d0 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fafac001b40 con 0x7fafb4142690 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 10 monclient(hunting): dump: 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:31.515 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.528+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fafa8001970 con 0x7fafb40b17d0 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 10 monclient(hunting): dump: 2024-03-29T11:06:31.516 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2201604014 0 0) 0x7fafb406feb0 con 0x7fafb4142690 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 >> v1:172.21.15.107:6790/0 conn(0x7fafb40b17d0 legacy=0x7fafb413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:31.517 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 >> v1:172.21.15.129:6789/0 conn(0x7fafb41422b0 legacy=0x7fafb4070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 1 monclient: found mon.a 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb406eea0 con 0x7fafb4142690 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafac001ca0 con 0x7fafb4142690 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafb8b02640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:06:31.518 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.529+0000 7fafb8b02640 10 monclient: _renew_subs 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafb8b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb414e580 con 0x7fafb4142690 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafb8b02640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafb8b02640 10 monclient: _renew_subs 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafb8b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb41df0a0 con 0x7fafb4142690 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafb8b02640 1 librados: init done 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafa2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fafac003b10 con 0x7fafb4142690 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafa2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:31.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafa2ffd640 10 monclient: dump: 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.530+0000 7fafb0d99640 4 set_mon_vals no callback set 2024-03-29T11:06:31.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.532+0000 7fafb8b02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:31.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.532+0000 7fafb8b02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:31.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.532+0000 7fafb8b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:31.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.532+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf98005080 con 0x7fafb4142690 2024-03-29T11:06:31.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.532+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fafac019ae0 con 0x7fafb4142690 2024-03-29T11:06:31.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.533+0000 7fafb259c640 10 monclient: get_auth_request con 0x7faf9406dff0 auth_method 0 2024-03-29T11:06:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.534+0000 7fafa2ffd640 10 client.4533.objecter ms_handle_connect 0x7faf9406dff0 2024-03-29T11:06:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.534+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fafac083af0 con 0x7fafb4142690 2024-03-29T11:06:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.534+0000 7fafa2ffd640 10 client.4533.objecter ms_dispatch 0x7fafb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.534+0000 7fafa2ffd640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.534+0000 7fafa2ffd640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.534+0000 7fafa2ffd640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T11:06:31.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.541+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafac04f200 con 0x7fafb4142690 2024-03-29T11:06:31.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.541+0000 7fafa2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:31.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.541+0000 7fafa2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:31.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.864+0000 7fafb8b02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:31.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.864+0000 7fafb8b02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:31.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.864+0000 7fafb8b02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:31.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.864+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf98005330 con 0x7fafb4142690 2024-03-29T11:06:31.847 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:06:31.847 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:31.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.867+0000 7fafa2ffd640 1 -- 172.21.15.107:0/1168180535 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fafac051300 con 0x7fafb4142690 2024-03-29T11:06:31.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.867+0000 7fafa2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:31.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.867+0000 7fafa2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:31.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.871+0000 7fafb8b02640 10 librados: watch_flush enter 2024-03-29T11:06:31.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.871+0000 7fafb8b02640 10 librados: watch_flush exit 2024-03-29T11:06:31.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.871+0000 7fafb8b02640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T11:06:31.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.871+0000 7fafb8b02640 10 client.4533.objecter successfully canceled tick 2024-03-29T11:06:31.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.871+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 >> v1:172.21.15.107:6816/882194755 conn(0x7faf9406dff0 legacy=0x7faf940704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:31.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.871+0000 7fafb8b02640 10 monclient: shutdown 2024-03-29T11:06:31.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.871+0000 7fafb8b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:31.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.871+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 >> v1:172.21.15.107:6789/0 conn(0x7fafb4142690 legacy=0x7fafb4070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:31.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.873+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 shutdown_connections 2024-03-29T11:06:31.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.873+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 >> 172.21.15.107:0/1168180535 conn(0x7fafb4066a00 msgr2=0x7fafb413ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:31.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.874+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 shutdown_connections 2024-03-29T11:06:31.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.874+0000 7fafb8b02640 1 -- 172.21.15.107:0/1168180535 wait complete. 2024-03-29T11:06:31.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:31.874+0000 7fafb8b02640 1 librados: shutdown 2024-03-29T11:06:31.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:38.676 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:38.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.846+0000 7f971b427640 10 monclient: get_monmap_and_config 2024-03-29T11:06:38.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.846+0000 7f971b427640 10 monclient: build_initial_monmap 2024-03-29T11:06:38.825 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.846+0000 7f971b427640 10 monclient: monmap: 2024-03-29T11:06:38.825 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:38.825 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:38.825 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:06:38.847183+0000 2024-03-29T11:06:38.825 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:06:38.847183+0000 2024-03-29T11:06:38.825 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:38.826 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:38.826 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:38.826 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:06:38.826 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:06:38.826 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:38.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.847+0000 7f971b427640 1 Processor -- start 2024-03-29T11:06:38.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.847+0000 7f971b427640 1 -- start start 2024-03-29T11:06:38.826 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.847+0000 7f971b427640 10 monclient: init 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 10 monclient(hunting): picked mon.noname-b con 0x7f97141422b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 10 monclient(hunting): picked mon.noname-a con 0x7f9714142690 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 10 monclient(hunting): picked mon.noname-c con 0x7f97140b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9714142c80 con 0x7f9714142690 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97140aae40 con 0x7f97140b17d0 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f971413fbd0 con 0x7f97141422b0 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f971b427640 10 monclient(hunting): authenticate will time out at 1330.116432s 2024-03-29T11:06:38.828 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f97191c3640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f97141422b0 0x7f971413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:40216/0 (socket says 172.21.15.107:40216) 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.848+0000 7f97189c2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9714142690 0x7f97140b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39672/0 (socket says 172.21.15.107:39672) 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f97191c3640 1 -- 172.21.15.107:0/1156084493 learned_addr learned my addr 172.21.15.107:0/1156084493 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665475406 0 0) 0x7f971413fbd0 con 0x7f97141422b0 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9700003520 con 0x7f97141422b0 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788803680 0 0) 0x7f9714142c80 con 0x7f9714142690 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:06:38.829 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f971413fbd0 con 0x7f9714142690 2024-03-29T11:06:38.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3698744005 0 0) 0x7f9700003520 con 0x7f97141422b0 2024-03-29T11:06:38.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.849+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 >> v1:172.21.15.129:6789/0 conn(0x7f97140b17d0 legacy=0x7f97140b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:06:38.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 >> v1:172.21.15.107:6789/0 conn(0x7f9714142690 legacy=0x7f97140b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:38.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:06:38.830 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 1 monclient: found mon.noname-b 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714151940 con 0x7f97141422b0 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 10 monclient: _finish_auth 0 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9704001690 con 0x7f97141422b0 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:38.831 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f970bfff640 10 monclient: dump: 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f971b427640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:06:38.832 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.851+0000 7f971b427640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:38.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.852+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9704001870 con 0x7f97141422b0 2024-03-29T11:06:38.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.852+0000 7f970bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:38.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.852+0000 7f971b427640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:38.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.852+0000 7f971b427640 4 set_mon_vals no callback set 2024-03-29T11:06:38.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.852+0000 7f970bfff640 1 -- 172.21.15.107:0/1156084493 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f9704001c40 con 0x7f97141422b0 2024-03-29T11:06:38.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.852+0000 7f970bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:38.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.852+0000 7f970bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:38.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.852+0000 7f970bfff640 10 monclient: dump: 2024-03-29T11:06:38.834 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.853+0000 7f971b427640 10 monclient: shutdown 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.853+0000 7f971b427640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.853+0000 7f971b427640 1 -- 172.21.15.107:0/1156084493 >> v1:172.21.15.107:6790/0 conn(0x7f97141422b0 legacy=0x7f971413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.853+0000 7f971b427640 1 -- 172.21.15.107:0/1156084493 shutdown_connections 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.853+0000 7f971b427640 1 -- 172.21.15.107:0/1156084493 >> 172.21.15.107:0/1156084493 conn(0x7f9714066a00 msgr2=0x7f9714066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.853+0000 7f971b427640 1 -- 172.21.15.107:0/1156084493 shutdown_connections 2024-03-29T11:06:38.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.854+0000 7f971b427640 1 -- 172.21.15.107:0/1156084493 wait complete. 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.855+0000 7f971b427640 10 monclient: build_initial_monmap 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.855+0000 7f971b427640 10 monclient: monmap: 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:06:38.836 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:38.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.855+0000 7f971b427640 1 librados: starting msgr at 2024-03-29T11:06:38.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.855+0000 7f971b427640 1 librados: starting objecter 2024-03-29T11:06:38.842 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.856+0000 7f971b427640 1 Processor -- start 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.856+0000 7f971b427640 1 -- start start 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.856+0000 7f971b427640 1 librados: setting wanted keys 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.856+0000 7f971b427640 1 librados: calling monclient init 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.856+0000 7f971b427640 10 monclient: init 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.857+0000 7f971b427640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.857+0000 7f971b427640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.857+0000 7f971b427640 10 monclient(hunting): picked mon.noname-b con 0x7f97140b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.857+0000 7f971b427640 10 monclient(hunting): picked mon.noname-c con 0x7f9714142690 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.857+0000 7f971b427640 10 monclient(hunting): picked mon.noname-a con 0x7f97140ab7b0 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.857+0000 7f971b427640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97140ac730 con 0x7f97140ab7b0 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.857+0000 7f97199c4640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f97140ab7b0 0x7f97141e1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40364/0 (socket says 172.21.15.107:40364) 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f97199c4640 1 -- 172.21.15.107:0/514262478 learned_addr learned my addr 172.21.15.107:0/514262478 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:38.843 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9714132b30 con 0x7f97140b17d0 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97140ac9a0 con 0x7f9714142690 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f971b427640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f971b427640 10 monclient(hunting): authenticate will time out at 1330.126476s 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 10 client.?.objecter ms_handle_connect 0x7f97140ab7b0 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203553226 0 0) 0x7f97140ac730 con 0x7f97140ab7b0 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f8004eb0 con 0x7f97140ab7b0 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 10 client.?.objecter ms_handle_connect 0x7f9714142690 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.858+0000 7f9709ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.859+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1799440250 0 0) 0x7f96f8004eb0 con 0x7f97140ab7b0 2024-03-29T11:06:38.844 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.859+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97140ac730 con 0x7f97140ab7b0 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.859+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f970c001b70 con 0x7f97140ab7b0 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.859+0000 7f9709ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.859+0000 7f9709ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.859+0000 7f9709ffb640 10 monclient(hunting): dump: 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:38.845 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.859+0000 7f9709ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.859+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1494310730 0 0) 0x7f97140ac730 con 0x7f97140ab7b0 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.859+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 >> v1:172.21.15.107:6790/0 conn(0x7f9714142690 legacy=0x7f9714130620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.860+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 >> v1:172.21.15.129:6789/0 conn(0x7f97140b17d0 legacy=0x7f971412ff10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f9709ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f9709ffb640 1 monclient: found mon.a 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f9709ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f9709ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97140acc80 con 0x7f97140ab7b0 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f9709ffb640 10 monclient: _finish_auth 0 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f9709ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f9709ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f971b427640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:06:38.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.861+0000 7f971b427640 10 monclient: _renew_subs 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.862+0000 7f971b427640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.862+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9714132510 con 0x7f97140ab7b0 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.862+0000 7f971b427640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.862+0000 7f971b427640 10 monclient: _renew_subs 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.862+0000 7f971b427640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.862+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97141e3410 con 0x7f97140ab7b0 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.862+0000 7f971b427640 1 librados: init done 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.863+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f970c002100 con 0x7f97140ab7b0 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.863+0000 7f9709ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.863+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f970c004e20 con 0x7f97140ab7b0 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.863+0000 7f9709ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.863+0000 7f9709ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.863+0000 7f9709ffb640 10 monclient: dump: 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:38.847 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:38.848 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:38.848 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:38.848 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:38.848 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:38.848 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:38.848 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:38.848 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:38.848 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:38.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.863+0000 7f970b7fe640 4 set_mon_vals no callback set 2024-03-29T11:06:38.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.864+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f970c01ae30 con 0x7f97140ab7b0 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.865+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f970c084e20 con 0x7f97140ab7b0 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.865+0000 7f9709ffb640 10 client.4539.objecter ms_dispatch 0x7f9714000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.865+0000 7f9709ffb640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.865+0000 7f9709ffb640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.865+0000 7f97191c3640 10 monclient: get_auth_request con 0x7f96f806dbd0 auth_method 0 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.865+0000 7f9709ffb640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.865+0000 7f9709ffb640 10 client.4539.objecter ms_handle_connect 0x7f96f806dbd0 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.866+0000 7f971b427640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.866+0000 7f971b427640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.866+0000 7f971b427640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:38.850 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.866+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f4005080 con 0x7f97140ab7b0 2024-03-29T11:06:38.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.874+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f970c050700 con 0x7f97140ab7b0 2024-03-29T11:06:38.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.874+0000 7f9709ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:38.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:38.874+0000 7f9709ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:39.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.178+0000 7f971b427640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:39.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.178+0000 7f971b427640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:39.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.178+0000 7f971b427640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:06:39.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.178+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96f4005330 con 0x7f97140ab7b0 2024-03-29T11:06:39.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.179+0000 7f9709ffb640 1 -- 172.21.15.107:0/514262478 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f970c052630 con 0x7f97140ab7b0 2024-03-29T11:06:39.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.179+0000 7f9709ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:39.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.179+0000 7f9709ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:39.158 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:06:39.158 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.182+0000 7f971b427640 10 librados: watch_flush enter 2024-03-29T11:06:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.182+0000 7f971b427640 10 librados: watch_flush exit 2024-03-29T11:06:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.182+0000 7f971b427640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T11:06:39.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.182+0000 7f971b427640 10 client.4539.objecter successfully canceled tick 2024-03-29T11:06:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.183+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 >> v1:172.21.15.107:6816/882194755 conn(0x7f96f806dbd0 legacy=0x7f96f8070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.183+0000 7f971b427640 10 monclient: shutdown 2024-03-29T11:06:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.183+0000 7f971b427640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:39.162 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.183+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 >> v1:172.21.15.107:6789/0 conn(0x7f97140ab7b0 legacy=0x7f97141e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.183+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 shutdown_connections 2024-03-29T11:06:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.183+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 >> 172.21.15.107:0/514262478 conn(0x7f9714066a00 msgr2=0x7f97140b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.183+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 shutdown_connections 2024-03-29T11:06:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.183+0000 7f971b427640 1 -- 172.21.15.107:0/514262478 wait complete. 2024-03-29T11:06:39.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:39.184+0000 7f971b427640 1 librados: shutdown 2024-03-29T11:06:39.177 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:45.984 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:46.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.149+0000 7f6fba29a640 10 monclient: get_monmap_and_config 2024-03-29T11:06:46.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.149+0000 7f6fba29a640 10 monclient: build_initial_monmap 2024-03-29T11:06:46.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.149+0000 7f6fba29a640 10 monclient: monmap: 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:06:46.150298+0000 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:06:46.150298+0000 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:06:46.129 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:46.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.150+0000 7f6fba29a640 1 Processor -- start 2024-03-29T11:06:46.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.150+0000 7f6fba29a640 1 -- start start 2024-03-29T11:06:46.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.150+0000 7f6fba29a640 10 monclient: init 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 10 monclient(hunting): picked mon.noname-c con 0x7f6fb40b0770 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb414a700 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 10 monclient(hunting): picked mon.noname-b con 0x7f6fb40ac5a0 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb40aba70 con 0x7f6fb414a700 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb40aecb0 con 0x7f6fb40b0770 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb40af150 con 0x7f6fb40ac5a0 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:46.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fba29a640 10 monclient(hunting): authenticate will time out at 1337.419565s 2024-03-29T11:06:46.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.151+0000 7f6fb8837640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f6fb40ac5a0 0x7f6fb4151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60764/0 (socket says 172.21.15.107:60764) 2024-03-29T11:06:46.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.152+0000 7f6fb8837640 1 -- 172.21.15.107:0/1383441564 learned_addr learned my addr 172.21.15.107:0/1383441564 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:46.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.152+0000 7f6fb37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6fb414a700 0x7f6fb40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47354/0 (socket says 172.21.15.107:47354) 2024-03-29T11:06:46.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.152+0000 7f6fb2ffd640 1 -- 172.21.15.107:0/1383441564 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729320539 0 0) 0x7f6fb40aba70 con 0x7f6fb414a700 2024-03-29T11:06:46.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.152+0000 7f6fb2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:46.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.152+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:46.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.152+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:46.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.152+0000 7f6fb2ffd640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:06:46.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.152+0000 7f6fb2ffd640 1 -- 172.21.15.107:0/1383441564 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa0003520 con 0x7f6fb414a700 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 1 -- 172.21.15.107:0/1383441564 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4271788978 0 0) 0x7f6fa0003520 con 0x7f6fb414a700 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 1 -- 172.21.15.107:0/1383441564 >> v1:172.21.15.107:6790/0 conn(0x7f6fb40ac5a0 legacy=0x7f6fb4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 1 -- 172.21.15.107:0/1383441564 >> v1:172.21.15.129:6789/0 conn(0x7f6fb40b0770 legacy=0x7f6fb4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 1 monclient: found mon.noname-a 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 1 -- 172.21.15.107:0/1383441564 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb40ac9d0 con 0x7f6fb414a700 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:06:46.133 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 1 -- 172.21.15.107:0/1383441564 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6f9c0015c0 con 0x7f6fb414a700 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: dump: 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:46.134 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 1 -- 172.21.15.107:0/1383441564 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f9c0017a0 con 0x7f6fb414a700 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 1 -- 172.21.15.107:0/1383441564 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6f9c001bb0 con 0x7f6fb414a700 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fb2ffd640 10 monclient: dump: 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:46.135 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:46.136 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:46.136 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:46.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fba29a640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:06:46.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fba29a640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:46.136 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.153+0000 7f6fba29a640 4 set_mon_vals no callback set 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.155+0000 7f6fba29a640 10 monclient: shutdown 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.155+0000 7f6fba29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.155+0000 7f6fba29a640 1 -- 172.21.15.107:0/1383441564 >> v1:172.21.15.107:6789/0 conn(0x7f6fb414a700 legacy=0x7f6fb40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.155+0000 7f6fba29a640 1 -- 172.21.15.107:0/1383441564 shutdown_connections 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.155+0000 7f6fba29a640 1 -- 172.21.15.107:0/1383441564 >> 172.21.15.107:0/1383441564 conn(0x7f6fb406b660 msgr2=0x7f6fb406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.155+0000 7f6fba29a640 1 -- 172.21.15.107:0/1383441564 shutdown_connections 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.155+0000 7f6fba29a640 1 -- 172.21.15.107:0/1383441564 wait complete. 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.156+0000 7f6fba29a640 10 monclient: build_initial_monmap 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.156+0000 7f6fba29a640 10 monclient: monmap: 2024-03-29T11:06:46.137 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.157+0000 7f6fba29a640 1 librados: starting msgr at 2024-03-29T11:06:46.138 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.157+0000 7f6fba29a640 1 librados: starting objecter 2024-03-29T11:06:46.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.157+0000 7f6fba29a640 1 Processor -- start 2024-03-29T11:06:46.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.157+0000 7f6fba29a640 1 -- start start 2024-03-29T11:06:46.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.157+0000 7f6fba29a640 1 librados: setting wanted keys 2024-03-29T11:06:46.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.157+0000 7f6fba29a640 1 librados: calling monclient init 2024-03-29T11:06:46.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.157+0000 7f6fba29a640 10 monclient: init 2024-03-29T11:06:46.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:46.145 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb40ac5a0 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 10 monclient(hunting): picked mon.noname-b con 0x7f6fb40b0770 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 10 monclient(hunting): picked mon.noname-c con 0x7f6fb4125200 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb4125bd0 con 0x7f6fb40ac5a0 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb41273a0 con 0x7f6fb40b0770 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb4144a10 con 0x7f6fb4125200 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fba29a640 10 monclient(hunting): authenticate will time out at 1337.426740s 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fb3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6fb40ac5a0 0x7f6fb41db5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47810/0 (socket says 172.21.15.107:47810) 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.158+0000 7f6fb3fff640 1 -- 172.21.15.107:0/3038653773 learned_addr learned my addr 172.21.15.107:0/3038653773 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.159+0000 7f6fb0ff9640 10 client.?.objecter ms_handle_connect 0x7f6fb4125200 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.159+0000 7f6fb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.159+0000 7f6fb0ff9640 10 client.?.objecter ms_handle_connect 0x7f6fb40ac5a0 2024-03-29T11:06:46.146 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.159+0000 7f6fb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992407458 0 0) 0x7f6fb4125bd0 con 0x7f6fb40ac5a0 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f94004eb0 con 0x7f6fb40ac5a0 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696979555 0 0) 0x7f6fb4144a10 con 0x7f6fb4125200 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb4125bd0 con 0x7f6fb4125200 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 client.?.objecter ms_handle_connect 0x7f6fb40b0770 2024-03-29T11:06:46.147 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270866563 0 0) 0x7f6fb41273a0 con 0x7f6fb40b0770 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb4144a10 con 0x7f6fb40b0770 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 500367734 0 0) 0x7f6f94004eb0 con 0x7f6fb40ac5a0 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fb41273a0 con 0x7f6fb40ac5a0 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6fa4001930 con 0x7f6fb40ac5a0 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.160+0000 7f6fb0ff9640 10 monclient(hunting): dump: 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.148 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636418843 0 0) 0x7f6fb4125bd0 con 0x7f6fb4125200 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f94004eb0 con 0x7f6fb4125200 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6fac001b20 con 0x7f6fb4125200 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient(hunting): dump: 2024-03-29T11:06:46.149 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244275331 0 0) 0x7f6fb4144a10 con 0x7f6fb40b0770 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fb4125bd0 con 0x7f6fb40b0770 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6f9c001bb0 con 0x7f6fb40b0770 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:46.150 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient(hunting): dump: 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3178990010 0 0) 0x7f6f94004eb0 con 0x7f6fb4125200 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 >> v1:172.21.15.129:6789/0 conn(0x7f6fb40b0770 legacy=0x7f6fb4143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 >> v1:172.21.15.107:6789/0 conn(0x7f6fb40ac5a0 legacy=0x7f6fb41db5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:46.151 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 monclient: found mon.c 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb4125840 con 0x7f6fb4125200 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fb0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fba29a640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fba29a640 10 monclient: _renew_subs 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fba29a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb41d9120 con 0x7f6fb4125200 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fba29a640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:46.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fba29a640 10 monclient: _renew_subs 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fba29a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb4126060 con 0x7f6fb4125200 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.161+0000 7f6fba29a640 1 librados: init done 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.162+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fac001f50 con 0x7f6fb4125200 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.162+0000 7f6fb0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.162+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6fac002100 con 0x7f6fb4125200 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.162+0000 7f6fb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.162+0000 7f6fb0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.162+0000 7f6fb0ff9640 10 monclient: dump: 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:46.153 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.163+0000 7f6fb27fc640 4 set_mon_vals no callback set 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.164+0000 7f6fba29a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.164+0000 7f6fba29a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.164+0000 7f6fba29a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.164+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f98005080 con 0x7f6fb4125200 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.164+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f6fac004d00 con 0x7f6fb4125200 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.165+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f6fac084b50 con 0x7f6fb4125200 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.165+0000 7f6fb0ff9640 10 client.4631.objecter ms_dispatch 0x7f6fb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:46.154 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.165+0000 7f6fb0ff9640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:46.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.165+0000 7f6fb0ff9640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:46.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.165+0000 7f6fb0ff9640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:06:46.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.165+0000 7f6fb3fff640 10 monclient: get_auth_request con 0x7f6f9406e060 auth_method 0 2024-03-29T11:06:46.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.166+0000 7f6fb0ff9640 10 client.4631.objecter ms_handle_connect 0x7f6f9406e060 2024-03-29T11:06:46.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.171+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fac050430 con 0x7f6fb4125200 2024-03-29T11:06:46.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.172+0000 7f6fb0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:46.155 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.172+0000 7f6fb0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:46.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.528+0000 7f6fba29a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:46.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.528+0000 7f6fba29a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:46.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.528+0000 7f6fba29a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:46.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.528+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f98005330 con 0x7f6fb4125200 2024-03-29T11:06:46.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.529+0000 7f6fb0ff9640 1 -- 172.21.15.107:0/3038653773 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f6fac052360 con 0x7f6fb4125200 2024-03-29T11:06:46.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.529+0000 7f6fb0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:46.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.529+0000 7f6fb0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:46.509 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:06:46.509 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:46.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.533+0000 7f6fba29a640 10 librados: watch_flush enter 2024-03-29T11:06:46.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.533+0000 7f6fba29a640 10 librados: watch_flush exit 2024-03-29T11:06:46.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.533+0000 7f6fba29a640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:06:46.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.533+0000 7f6fba29a640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:06:46.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.534+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 >> v1:172.21.15.107:6816/882194755 conn(0x7f6f9406e060 legacy=0x7f6f94070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:46.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.534+0000 7f6fba29a640 10 monclient: shutdown 2024-03-29T11:06:46.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.534+0000 7f6fba29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:46.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.534+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 >> v1:172.21.15.107:6790/0 conn(0x7f6fb4125200 legacy=0x7f6fb4144300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:46.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.534+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 shutdown_connections 2024-03-29T11:06:46.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.534+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 >> 172.21.15.107:0/3038653773 conn(0x7f6fb406b660 msgr2=0x7f6fb40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:46.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.534+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 shutdown_connections 2024-03-29T11:06:46.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.534+0000 7f6fba29a640 1 -- 172.21.15.107:0/3038653773 wait complete. 2024-03-29T11:06:46.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:46.534+0000 7f6fba29a640 1 librados: shutdown 2024-03-29T11:06:46.527 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:53.335 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:53.495 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.516+0000 7f3c89325640 10 monclient: get_monmap_and_config 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.516+0000 7f3c89325640 10 monclient: build_initial_monmap 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.516+0000 7f3c89325640 10 monclient: monmap: 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:06:53.517620+0000 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:06:53.517620+0000 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:06:53.496 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:53.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.517+0000 7f3c89325640 1 Processor -- start 2024-03-29T11:06:53.497 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.517+0000 7f3c89325640 1 -- start start 2024-03-29T11:06:53.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.517+0000 7f3c89325640 10 monclient: init 2024-03-29T11:06:53.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:53.498 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:06:53.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 10 monclient(hunting): picked mon.noname-c con 0x7f3c840b46b0 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:53.502 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 10 monclient(hunting): picked mon.noname-b con 0x7f3c840aeae0 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 10 monclient(hunting): picked mon.noname-a con 0x7f3c8413fa10 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c840b06a0 con 0x7f3c8413fa10 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c840b4230 con 0x7f3c840b46b0 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c840b0910 con 0x7f3c840aeae0 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.518+0000 7f3c89325640 10 monclient(hunting): authenticate will time out at 1344.786791s 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.519+0000 7f3c827fc640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3c840aeae0 0x7f3c8413f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54292/0 (socket says 172.21.15.107:54292) 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.519+0000 7f3c827fc640 1 -- 172.21.15.107:0/2813708782 learned_addr learned my addr 172.21.15.107:0/2813708782 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.519+0000 7f3c827fc640 1 -- 172.21.15.107:0/2813708782 >> v1:172.21.15.107:6790/0 conn(0x7f3c840aeae0 legacy=0x7f3c8413f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.519+0000 7f3c827fc640 1 -- 172.21.15.107:0/2813708782 >> v1:172.21.15.107:6790/0 conn(0x7f3c840aeae0 legacy=0x7f3c8413f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.519+0000 7f3c827fc640 1 --1- 172.21.15.107:0/2813708782 >> v1:172.21.15.107:6790/0 conn(0x7f3c840aeae0 0x7f3c8413f300 :-1 s=OPENED pgs=1008061 cs=1 l=1).handle_message read tag failed 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.519+0000 7f3c827fc640 1 --1- 172.21.15.107:0/2813708782 >> v1:172.21.15.107:6790/0 conn(0x7f3c840aeae0 0x7f3c8413f300 :-1 s=OPENED pgs=1008061 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:06:53.503 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.519+0000 7f3c81ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.107:6790/0 2024-03-29T11:06:53.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876737869 0 0) 0x7f3c840b06a0 con 0x7f3c8413fa10 2024-03-29T11:06:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:06:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:06:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c6c003520 con 0x7f3c8413fa10 2024-03-29T11:06:53.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948323026 0 0) 0x7f3c840b4230 con 0x7f3c840b46b0 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.526+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c840b06a0 con 0x7f3c840b46b0 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.527+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661840040 0 0) 0x7f3c6c003520 con 0x7f3c8413fa10 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.527+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 >> v1:172.21.15.107:6790/0 conn(0x7f3c840aeae0 legacy=0x7f3c8413f300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.527+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 >> v1:172.21.15.129:6789/0 conn(0x7f3c840b46b0 legacy=0x7f3c840b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.527+0000 7f3c81ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.527+0000 7f3c81ffb640 1 monclient: found mon.noname-a 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c840af6d0 con 0x7f3c8413fa10 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 10 monclient: _finish_auth 0 2024-03-29T11:06:53.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3c7c001a10 con 0x7f3c8413fa10 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c81ffb640 10 monclient: dump: 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:53.509 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:53.510 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:53.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c89325640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:06:53.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.528+0000 7f3c89325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:53.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.531+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c7c001b70 con 0x7f3c8413fa10 2024-03-29T11:06:53.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.531+0000 7f3c81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:53.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.531+0000 7f3c81ffb640 1 -- 172.21.15.107:0/2813708782 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3c7c001fc0 con 0x7f3c8413fa10 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.531+0000 7f3c81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.531+0000 7f3c81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.531+0000 7f3c81ffb640 10 monclient: dump: 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:53.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.531+0000 7f3c89325640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:53.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.531+0000 7f3c89325640 4 set_mon_vals no callback set 2024-03-29T11:06:53.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.533+0000 7f3c89325640 10 monclient: shutdown 2024-03-29T11:06:53.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.533+0000 7f3c89325640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:53.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.533+0000 7f3c89325640 1 -- 172.21.15.107:0/2813708782 >> v1:172.21.15.107:6789/0 conn(0x7f3c8413fa10 legacy=0x7f3c8414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:53.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.533+0000 7f3c89325640 1 -- 172.21.15.107:0/2813708782 shutdown_connections 2024-03-29T11:06:53.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.533+0000 7f3c89325640 1 -- 172.21.15.107:0/2813708782 >> 172.21.15.107:0/2813708782 conn(0x7f3c84066a00 msgr2=0x7f3c84066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:53.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.535+0000 7f3c89325640 1 -- 172.21.15.107:0/2813708782 shutdown_connections 2024-03-29T11:06:53.515 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.536+0000 7f3c89325640 1 -- 172.21.15.107:0/2813708782 wait complete. 2024-03-29T11:06:53.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.537+0000 7f3c89325640 10 monclient: build_initial_monmap 2024-03-29T11:06:53.519 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.537+0000 7f3c89325640 10 monclient: monmap: 2024-03-29T11:06:53.519 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:06:53.519 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:53.519 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:06:53.519 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:06:53.519 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:53.519 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:53.519 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.537+0000 7f3c89325640 1 librados: starting msgr at 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.537+0000 7f3c89325640 1 librados: starting objecter 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.538+0000 7f3c89325640 1 Processor -- start 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.538+0000 7f3c89325640 1 -- start start 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.538+0000 7f3c89325640 1 librados: setting wanted keys 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.538+0000 7f3c89325640 1 librados: calling monclient init 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.538+0000 7f3c89325640 10 monclient: init 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 10 monclient(hunting): picked mon.noname-a con 0x7f3c840aeae0 addr v1:172.21.15.107:6789/0 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 10 monclient(hunting): picked mon.noname-c con 0x7f3c840b46b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:06:53.520 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 10 monclient(hunting): picked mon.noname-b con 0x7f3c8413fa10 addr v1:172.21.15.129:6789/0 2024-03-29T11:06:53.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c84142b70 con 0x7f3c840aeae0 2024-03-29T11:06:53.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c841425b0 con 0x7f3c8413fa10 2024-03-29T11:06:53.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c84142740 con 0x7f3c840b46b0 2024-03-29T11:06:53.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:53.521 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.539+0000 7f3c89325640 10 monclient(hunting): authenticate will time out at 1344.807082s 2024-03-29T11:06:53.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c827fc640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3c840b46b0 0x7f3c84134c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:55168/0 (socket says 172.21.15.107:55168) 2024-03-29T11:06:53.522 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c827fc640 1 -- 172.21.15.107:0/3166348606 learned_addr learned my addr 172.21.15.107:0/3166348606 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c6b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c840b46b0 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773502316 0 0) 0x7f3c84142740 con 0x7f3c840b46b0 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c6b7fe640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.541+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c70004eb0 con 0x7f3c840b46b0 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3782221617 0 0) 0x7f3c70004eb0 con 0x7f3c840b46b0 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c84142740 con 0x7f3c840b46b0 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3c740018f0 con 0x7f3c840b46b0 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:53.523 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 10 monclient(hunting): dump: 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2025315889 0 0) 0x7f3c84142740 con 0x7f3c840b46b0 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 >> v1:172.21.15.129:6789/0 conn(0x7f3c8413fa10 legacy=0x7f3c841e3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:53.524 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.542+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 >> v1:172.21.15.107:6789/0 conn(0x7f3c840aeae0 legacy=0x7f3c84141ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c6b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c6b7fe640 1 monclient: found mon.c 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c6b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c841428d0 con 0x7f3c840b46b0 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c6b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c6b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c89325640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c89325640 10 monclient: _renew_subs 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c89325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:53.526 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c84135320 con 0x7f3c840b46b0 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c89325640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c89325640 10 monclient: _renew_subs 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c89325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8413bcf0 con 0x7f3c840b46b0 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.544+0000 7f3c89325640 1 librados: init done 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.545+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c74001a50 con 0x7f3c840b46b0 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.545+0000 7f3c6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.545+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3c74004ac0 con 0x7f3c840b46b0 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.545+0000 7f3c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.545+0000 7f3c6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.545+0000 7f3c6b7fe640 10 monclient: dump: 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:06:53.527 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:53.528 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:06:53.528 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:53.528 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:06:53.528 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:06:53.528 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:06:53.528 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:06:53.528 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:06:53.528 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.546+0000 7f3c817fa640 4 set_mon_vals no callback set 2024-03-29T11:06:53.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.549+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f3c74004c20 con 0x7f3c840b46b0 2024-03-29T11:06:53.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.550+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f3c74084c20 con 0x7f3c840b46b0 2024-03-29T11:06:53.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.550+0000 7f3c6b7fe640 10 client.4634.objecter ms_dispatch 0x7f3c84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:53.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.550+0000 7f3c6b7fe640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:53.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.550+0000 7f3c6b7fe640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:53.529 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.550+0000 7f3c6b7fe640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:06:53.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.551+0000 7f3c82ffd640 10 monclient: get_auth_request con 0x7f3c7006dd10 auth_method 0 2024-03-29T11:06:53.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.551+0000 7f3c89325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:53.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.551+0000 7f3c89325640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:53.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.551+0000 7f3c89325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:53.530 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.551+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c64005080 con 0x7f3c840b46b0 2024-03-29T11:06:53.531 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.551+0000 7f3c6b7fe640 10 client.4634.objecter ms_handle_connect 0x7f3c7006dd10 2024-03-29T11:06:53.540 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.561+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c74050330 con 0x7f3c840b46b0 2024-03-29T11:06:53.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.562+0000 7f3c6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:53.541 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.562+0000 7f3c6b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:53.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.900+0000 7f3c89325640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:53.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.900+0000 7f3c89325640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:53.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.900+0000 7f3c89325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:06:53.880 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.900+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c64005330 con 0x7f3c840b46b0 2024-03-29T11:06:53.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.903+0000 7f3c6b7fe640 1 -- 172.21.15.107:0/3166348606 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f3c74052430 con 0x7f3c840b46b0 2024-03-29T11:06:53.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.903+0000 7f3c6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:53.882 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.903+0000 7f3c6b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:53.882 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:06:53.883 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:53.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.907+0000 7f3c89325640 10 librados: watch_flush enter 2024-03-29T11:06:53.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.907+0000 7f3c89325640 10 librados: watch_flush exit 2024-03-29T11:06:53.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.907+0000 7f3c89325640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:06:53.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.907+0000 7f3c89325640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:06:53.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.907+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 >> v1:172.21.15.107:6816/882194755 conn(0x7f3c7006dd10 legacy=0x7f3c700701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:53.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.907+0000 7f3c89325640 10 monclient: shutdown 2024-03-29T11:06:53.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.907+0000 7f3c89325640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:53.886 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.907+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 >> v1:172.21.15.107:6790/0 conn(0x7f3c840b46b0 legacy=0x7f3c84134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:53.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.908+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 shutdown_connections 2024-03-29T11:06:53.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.908+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 >> 172.21.15.107:0/3166348606 conn(0x7f3c84066a00 msgr2=0x7f3c8413f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:53.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.908+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 shutdown_connections 2024-03-29T11:06:53.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.908+0000 7f3c89325640 1 -- 172.21.15.107:0/3166348606 wait complete. 2024-03-29T11:06:53.887 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:06:53.908+0000 7f3c89325640 1 librados: shutdown 2024-03-29T11:06:53.904 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:00.711 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.866+0000 7fc4378c1640 10 monclient: get_monmap_and_config 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.866+0000 7fc4378c1640 10 monclient: build_initial_monmap 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.866+0000 7fc4378c1640 10 monclient: monmap: 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:07:00.867078+0000 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:07:00.867078+0000 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:07:00.845 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:07:00.846 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:00.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.866+0000 7fc4378c1640 1 Processor -- start 2024-03-29T11:07:00.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.867+0000 7fc4378c1640 1 -- start start 2024-03-29T11:07:00.846 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.867+0000 7fc4378c1640 10 monclient: init 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 10 monclient(hunting): picked mon.noname-a con 0x7fc4301422b0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 10 monclient(hunting): picked mon.noname-c con 0x7fc430142690 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 10 monclient(hunting): picked mon.noname-b con 0x7fc4300b17d0 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc430142c80 con 0x7fc4301422b0 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4300aae40 con 0x7fc430142690 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc435e5e640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fc4300b17d0 0x7fc4300b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48224/0 (socket says 172.21.15.107:48224) 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc435e5e640 1 -- 172.21.15.107:0/353391888 learned_addr learned my addr 172.21.15.107:0/353391888 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:00.848 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 1 -- 172.21.15.107:0/353391888 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43013fbd0 con 0x7fc4300b17d0 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc4378c1640 10 monclient(hunting): authenticate will time out at 1352.136448s 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc427fff640 1 -- 172.21.15.107:0/353391888 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824071321 0 0) 0x7fc43013fbd0 con 0x7fc4300b17d0 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc427fff640 10 monclient(hunting): _negotiate 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc427fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc427fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc427fff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.868+0000 7fc427fff640 1 -- 172.21.15.107:0/353391888 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c003520 con 0x7fc4300b17d0 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.869+0000 7fc427fff640 1 -- 172.21.15.107:0/353391888 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1240180972 0 0) 0x7fc41c003520 con 0x7fc4300b17d0 2024-03-29T11:07:00.849 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.869+0000 7fc427fff640 1 -- 172.21.15.107:0/353391888 >> v1:172.21.15.129:6789/0 conn(0x7fc430142690 legacy=0x7fc4300b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:07:00.851 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 1 -- 172.21.15.107:0/353391888 >> v1:172.21.15.107:6789/0 conn(0x7fc4301422b0 legacy=0x7fc43013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:00.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: _finish_hunting 0 2024-03-29T11:07:00.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 1 monclient: found mon.noname-b 2024-03-29T11:07:00.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:00.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:07:00.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 1 -- 172.21.15.107:0/353391888 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc430151940 con 0x7fc4300b17d0 2024-03-29T11:07:00.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: _finish_auth 0 2024-03-29T11:07:00.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: _check_auth_tickets 2024-03-29T11:07:00.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:00.853 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 1 -- 172.21.15.107:0/353391888 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc428001a10 con 0x7fc4300b17d0 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: dump: 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:00.854 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc4378c1640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc4378c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 1 -- 172.21.15.107:0/353391888 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc428001b70 con 0x7fc4300b17d0 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 1 -- 172.21.15.107:0/353391888 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc428001f80 con 0x7fc4300b17d0 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.872+0000 7fc427fff640 10 monclient: dump: 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:00.855 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:00.856 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:00.856 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:00.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.873+0000 7fc4378c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:00.861 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.873+0000 7fc4378c1640 4 set_mon_vals no callback set 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.873+0000 7fc4378c1640 10 monclient: shutdown 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.873+0000 7fc4378c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.873+0000 7fc4378c1640 1 -- 172.21.15.107:0/353391888 >> v1:172.21.15.107:6790/0 conn(0x7fc4300b17d0 legacy=0x7fc4300b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.874+0000 7fc4378c1640 1 -- 172.21.15.107:0/353391888 shutdown_connections 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.874+0000 7fc4378c1640 1 -- 172.21.15.107:0/353391888 >> 172.21.15.107:0/353391888 conn(0x7fc430066a00 msgr2=0x7fc430066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.874+0000 7fc4378c1640 1 -- 172.21.15.107:0/353391888 shutdown_connections 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.874+0000 7fc4378c1640 1 -- 172.21.15.107:0/353391888 wait complete. 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.875+0000 7fc4378c1640 10 monclient: build_initial_monmap 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.875+0000 7fc4378c1640 10 monclient: monmap: 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:07:00.862 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.875+0000 7fc4378c1640 1 librados: starting msgr at 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.875+0000 7fc4378c1640 1 librados: starting objecter 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.875+0000 7fc4378c1640 1 Processor -- start 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc4378c1640 1 -- start start 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc4378c1640 1 librados: setting wanted keys 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc4378c1640 1 librados: calling monclient init 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc4378c1640 10 monclient: init 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc4378c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc4378c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:07:00.863 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc4378c1640 10 monclient(hunting): picked mon.noname-b con 0x7fc4300b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc434e5c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc4301422b0 0x7fc4300b7e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57860/0 (socket says 172.21.15.107:57860) 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc434e5c640 1 -- 172.21.15.107:0/3372584226 learned_addr learned my addr 172.21.15.107:0/3372584226 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.876+0000 7fc4378c1640 10 monclient(hunting): picked mon.noname-a con 0x7fc4301422b0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.877+0000 7fc4378c1640 10 monclient(hunting): picked mon.noname-c con 0x7fc430142690 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.877+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43006de90 con 0x7fc4301422b0 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.877+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43006e020 con 0x7fc4300b17d0 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.877+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc430131980 con 0x7fc430142690 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.877+0000 7fc4378c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.877+0000 7fc4378c1640 10 monclient(hunting): authenticate will time out at 1352.145423s 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.878+0000 7fc425ffb640 10 client.?.objecter ms_handle_connect 0x7fc430142690 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.878+0000 7fc425ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.878+0000 7fc425ffb640 10 client.?.objecter ms_handle_connect 0x7fc4301422b0 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.878+0000 7fc425ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:00.864 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.878+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042889503 0 0) 0x7fc430131980 con 0x7fc430142690 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc414004eb0 con 0x7fc430142690 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56653532 0 0) 0x7fc43006de90 con 0x7fc4301422b0 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc430131980 con 0x7fc4301422b0 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4227302099 0 0) 0x7fc414004eb0 con 0x7fc430142690 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc43006de90 con 0x7fc430142690 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320702952 0 0) 0x7fc430131980 con 0x7fc4301422b0 2024-03-29T11:07:00.865 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc414004eb0 con 0x7fc4301422b0 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc428001e80 con 0x7fc430142690 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): dump: 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:00.866 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc42c001640 con 0x7fc4301422b0 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.879+0000 7fc425ffb640 10 monclient(hunting): dump: 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:00.867 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1279125552 0 0) 0x7fc43006de90 con 0x7fc430142690 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 >> v1:172.21.15.129:6789/0 conn(0x7fc4300b17d0 legacy=0x7fc4300b7020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 >> v1:172.21.15.107:6789/0 conn(0x7fc4301422b0 legacy=0x7fc4300b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 1 monclient: found mon.c 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc430131b10 con 0x7fc430142690 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 10 monclient: _finish_auth 0 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc425ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc4378c1640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc4378c1640 10 monclient: _renew_subs 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc4378c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:00.868 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.880+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4300b5420 con 0x7fc430142690 2024-03-29T11:07:00.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.881+0000 7fc4378c1640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:00.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.881+0000 7fc4378c1640 10 monclient: _renew_subs 2024-03-29T11:07:00.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.881+0000 7fc4378c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:00.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.881+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4300b58b0 con 0x7fc430142690 2024-03-29T11:07:00.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.881+0000 7fc4378c1640 1 librados: init done 2024-03-29T11:07:00.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.882+0000 7fc4378c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:00.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.882+0000 7fc4378c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:00.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.883+0000 7fc4378c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:00.869 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.883+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc430131d00 con 0x7fc430142690 2024-03-29T11:07:00.870 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.883+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc428001fe0 con 0x7fc430142690 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.883+0000 7fc425ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.883+0000 7fc4277fe640 4 set_mon_vals no callback set 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.883+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fc428004fb0 con 0x7fc430142690 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.883+0000 7fc425ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.883+0000 7fc425ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.883+0000 7fc425ffb640 10 monclient: dump: 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:00.871 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.884+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fc42801af80 con 0x7fc430142690 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.885+0000 7fc43565d640 10 monclient: get_auth_request con 0x7fc41406ded0 auth_method 0 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.885+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7fc428084ef0 con 0x7fc430142690 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.885+0000 7fc425ffb640 10 client.4640.objecter ms_dispatch 0x7fc430000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.885+0000 7fc425ffb640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.885+0000 7fc425ffb640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.885+0000 7fc425ffb640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:07:00.872 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.885+0000 7fc425ffb640 10 client.4640.objecter ms_handle_connect 0x7fc41406ded0 2024-03-29T11:07:00.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.892+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4280507d0 con 0x7fc430142690 2024-03-29T11:07:00.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.893+0000 7fc425ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:00.873 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:00.893+0000 7fc425ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:01.182 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.203+0000 7fc4378c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:01.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.203+0000 7fc4378c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:01.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.203+0000 7fc4378c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:01.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.203+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc430075f10 con 0x7fc430142690 2024-03-29T11:07:01.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.204+0000 7fc425ffb640 1 -- 172.21.15.107:0/3372584226 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7fc428052700 con 0x7fc430142690 2024-03-29T11:07:01.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.204+0000 7fc425ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:01.183 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.204+0000 7fc425ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:01.183 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:07:01.184 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.206+0000 7fc4378c1640 10 librados: watch_flush enter 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.206+0000 7fc4378c1640 10 librados: watch_flush exit 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.206+0000 7fc4378c1640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 >> v1:172.21.15.107:6816/882194755 conn(0x7fc41406ded0 legacy=0x7fc414070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 10 monclient: shutdown 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 >> v1:172.21.15.107:6790/0 conn(0x7fc430142690 legacy=0x7fc4300712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 shutdown_connections 2024-03-29T11:07:01.186 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 >> 172.21.15.107:0/3372584226 conn(0x7fc430066a00 msgr2=0x7fc4300b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:01.187 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 shutdown_connections 2024-03-29T11:07:01.187 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 1 -- 172.21.15.107:0/3372584226 wait complete. 2024-03-29T11:07:01.187 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:01.207+0000 7fc4378c1640 1 librados: shutdown 2024-03-29T11:07:01.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:08.007 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:08.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.169+0000 7f1c6cb8f640 10 monclient: get_monmap_and_config 2024-03-29T11:07:08.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.169+0000 7f1c6cb8f640 10 monclient: build_initial_monmap 2024-03-29T11:07:08.149 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.169+0000 7f1c6cb8f640 10 monclient: monmap: 2024-03-29T11:07:08.149 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:08.149 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:08.149 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:07:08.170598+0000 2024-03-29T11:07:08.149 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:07:08.170598+0000 2024-03-29T11:07:08.150 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:08.150 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:08.150 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:08.150 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:07:08.150 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:07:08.150 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:08.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.171+0000 7f1c6cb8f640 1 Processor -- start 2024-03-29T11:07:08.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.171+0000 7f1c6cb8f640 1 -- start start 2024-03-29T11:07:08.152 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.171+0000 7f1c6cb8f640 10 monclient: init 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 10 monclient(hunting): picked mon.noname-c con 0x7f1c680a9ac0 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6813ee90 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 10 monclient(hunting): picked mon.noname-a con 0x7f1c680af900 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c680afd10 con 0x7f1c680af900 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c680af6f0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c680aac80 con 0x7f1c6813ee90 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:08.153 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.172+0000 7f1c6cb8f640 10 monclient(hunting): authenticate will time out at 1359.440300s 2024-03-29T11:07:08.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.175+0000 7f1c65d9b640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f1c6813ee90 0x7f1c6813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33213/0 (socket says 172.21.15.107:33213) 2024-03-29T11:07:08.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.175+0000 7f1c65d9b640 1 -- 172.21.15.107:0/867818008 learned_addr learned my addr 172.21.15.107:0/867818008 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:08.157 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523397125 0 0) 0x7f1c680af6f0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 10 monclient(hunting): _negotiate 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c50003500 con 0x7f1c680a9ac0 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532782022 0 0) 0x7f1c680aac80 con 0x7f1c6813ee90 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 10 monclient(hunting): _negotiate 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c680af6f0 con 0x7f1c6813ee90 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048301087 0 0) 0x7f1c50003500 con 0x7f1c680a9ac0 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 >> v1:172.21.15.107:6790/0 conn(0x7f1c6813ee90 legacy=0x7f1c6813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:08.158 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.177+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 >> v1:172.21.15.107:6789/0 conn(0x7f1c680af900 legacy=0x7f1c68149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: _finish_hunting 0 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 1 monclient: found mon.noname-c 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c680aaf60 con 0x7f1c680a9ac0 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: _finish_auth 0 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: _check_auth_tickets 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1c58001690 con 0x7f1c680a9ac0 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: dump: 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:08.159 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6cb8f640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6cb8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c580018b0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 1 -- 172.21.15.107:0/867818008 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1c58001c40 con 0x7f1c680a9ac0 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:08.160 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6559a640 10 monclient: dump: 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6cb8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:08.161 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.178+0000 7f1c6cb8f640 4 set_mon_vals no callback set 2024-03-29T11:07:08.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.179+0000 7f1c6cb8f640 10 monclient: shutdown 2024-03-29T11:07:08.163 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.179+0000 7f1c6cb8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.179+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/867818008 >> v1:172.21.15.129:6789/0 conn(0x7f1c680a9ac0 legacy=0x7f1c680adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.179+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/867818008 shutdown_connections 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.179+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/867818008 >> 172.21.15.107:0/867818008 conn(0x7f1c680b35e0 msgr2=0x7f1c680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.179+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/867818008 shutdown_connections 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.179+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/867818008 wait complete. 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.181+0000 7f1c6cb8f640 10 monclient: build_initial_monmap 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.181+0000 7f1c6cb8f640 10 monclient: monmap: 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:08.164 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.181+0000 7f1c6cb8f640 1 librados: starting msgr at 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.182+0000 7f1c6cb8f640 1 librados: starting objecter 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.182+0000 7f1c6cb8f640 1 Processor -- start 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.182+0000 7f1c6cb8f640 1 -- start start 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.182+0000 7f1c6cb8f640 1 librados: setting wanted keys 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.182+0000 7f1c6cb8f640 1 librados: calling monclient init 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.182+0000 7f1c6cb8f640 10 monclient: init 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 10 monclient(hunting): picked mon.noname-b con 0x7f1c680a9ac0 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 10 monclient(hunting): picked mon.noname-c con 0x7f1c680af900 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:08.165 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 10 monclient(hunting): picked mon.noname-a con 0x7f1c6813ee90 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6806f540 con 0x7f1c6813ee90 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c68142520 con 0x7f1c680a9ac0 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c681421f0 con 0x7f1c680af900 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.183+0000 7f1c6cb8f640 10 monclient(hunting): authenticate will time out at 1359.451552s 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c6659c640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7f1c680a9ac0 0x7f1c681396f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:37798/0 (socket says 172.21.15.107:37798) 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c6659c640 1 -- 172.21.15.107:0/1639529282 learned_addr learned my addr 172.21.15.107:0/1639529282 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c56ffd640 10 client.?.objecter ms_handle_connect 0x7f1c680a9ac0 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c56ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205656057 0 0) 0x7f1c68142520 con 0x7f1c680a9ac0 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c56ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:08.166 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c56ffd640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:07:08.167 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.184+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c48004eb0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.185+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2604266191 0 0) 0x7f1c48004eb0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.185+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c68142520 con 0x7f1c680a9ac0 2024-03-29T11:07:08.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.185+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1c58001cc0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.185+0000 7f1c56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:08.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.185+0000 7f1c56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:08.170 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.185+0000 7f1c56ffd640 10 monclient(hunting): dump: 2024-03-29T11:07:08.170 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:08.170 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:08.170 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.185+0000 7f1c56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.186+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1221991035 0 0) 0x7f1c68142520 con 0x7f1c680a9ac0 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.186+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 >> v1:172.21.15.107:6790/0 conn(0x7f1c680af900 legacy=0x7f1c681d66a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.186+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 >> v1:172.21.15.107:6789/0 conn(0x7f1c6813ee90 legacy=0x7f1c681de770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.187+0000 7f1c56ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.188+0000 7f1c56ffd640 1 monclient: found mon.b 2024-03-29T11:07:08.171 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.188+0000 7f1c56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.188+0000 7f1c56ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.188+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6806ecb0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.188+0000 7f1c56ffd640 10 monclient: _finish_auth 0 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.188+0000 7f1c56ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.188+0000 7f1c56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.188+0000 7f1c6cb8f640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.188+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c58003910 con 0x7f1c680a9ac0 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c6cb8f640 10 monclient: _renew_subs 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c6cb8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6813f800 con 0x7f1c680a9ac0 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c6cb8f640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1c58003d60 con 0x7f1c680a9ac0 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:08.172 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c56ffd640 10 monclient: dump: 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c6cb8f640 10 monclient: _renew_subs 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c6cb8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c68139e00 con 0x7f1c680a9ac0 2024-03-29T11:07:08.173 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c6cb8f640 1 librados: init done 2024-03-29T11:07:08.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.189+0000 7f1c64d99640 4 set_mon_vals no callback set 2024-03-29T11:07:08.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.191+0000 7f1c6cb8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:08.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.191+0000 7f1c6cb8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:08.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.191+0000 7f1c6cb8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:08.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.191+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c68070660 con 0x7f1c680a9ac0 2024-03-29T11:07:08.174 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.192+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f1c58003ec0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.193+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f1c58084c90 con 0x7f1c680a9ac0 2024-03-29T11:07:08.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.194+0000 7f1c65d9b640 10 monclient: get_auth_request con 0x7f1c4806dc90 auth_method 0 2024-03-29T11:07:08.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.194+0000 7f1c56ffd640 10 client.4516.objecter ms_dispatch 0x7f1c68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:08.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.194+0000 7f1c56ffd640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:08.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.194+0000 7f1c56ffd640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:08.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.194+0000 7f1c56ffd640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T11:07:08.175 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.195+0000 7f1c56ffd640 10 client.4516.objecter ms_handle_connect 0x7f1c4806dc90 2024-03-29T11:07:08.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.200+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c58050570 con 0x7f1c680a9ac0 2024-03-29T11:07:08.179 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.200+0000 7f1c56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:08.180 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.200+0000 7f1c56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:08.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.507+0000 7f1c6cb8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:08.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.507+0000 7f1c6cb8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:08.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.507+0000 7f1c6cb8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:08.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.507+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c6813bcd0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.508+0000 7f1c56ffd640 1 -- 172.21.15.107:0/1639529282 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3154165527) 0x7f1c580524a0 con 0x7f1c680a9ac0 2024-03-29T11:07:08.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.508+0000 7f1c56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:08.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.508+0000 7f1c56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:08.489 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:07:08.489 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T10:57:12.857130+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:12.857130+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.512+0000 7f1c6cb8f640 10 librados: watch_flush enter 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.512+0000 7f1c6cb8f640 10 librados: watch_flush exit 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.512+0000 7f1c6cb8f640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.512+0000 7f1c6cb8f640 10 client.4516.objecter successfully canceled tick 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.512+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 >> v1:172.21.15.107:6816/882194755 conn(0x7f1c4806dc90 legacy=0x7f1c48070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.513+0000 7f1c6cb8f640 10 monclient: shutdown 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.513+0000 7f1c6cb8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.513+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 >> v1:172.21.15.129:6789/0 conn(0x7f1c680a9ac0 legacy=0x7f1c681396f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.513+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 shutdown_connections 2024-03-29T11:07:08.493 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.513+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 >> 172.21.15.107:0/1639529282 conn(0x7f1c680b35e0 msgr2=0x7f1c6813b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:08.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.513+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 shutdown_connections 2024-03-29T11:07:08.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.513+0000 7f1c6cb8f640 1 -- 172.21.15.107:0/1639529282 wait complete. 2024-03-29T11:07:08.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:08.513+0000 7f1c6cb8f640 1 librados: shutdown 2024-03-29T11:07:08.508 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:15.315 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:15.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.466+0000 7f23d470e640 10 monclient: get_monmap_and_config 2024-03-29T11:07:15.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.466+0000 7f23d470e640 10 monclient: build_initial_monmap 2024-03-29T11:07:15.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.466+0000 7f23d470e640 10 monclient: monmap: 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:07:15.467292+0000 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:07:15.467292+0000 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.467+0000 7f23d470e640 1 Processor -- start 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.467+0000 7f23d470e640 1 -- start start 2024-03-29T11:07:15.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.467+0000 7f23d470e640 10 monclient: init 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 10 monclient(hunting): picked mon.noname-c con 0x7f23cc0ad6c0 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 10 monclient(hunting): picked mon.noname-b con 0x7f23cc0af6c0 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 10 monclient(hunting): picked mon.noname-a con 0x7f23cc14fdc0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0a94e0 con 0x7f23cc14fdc0 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0af3a0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0afb60 con 0x7f23cc0af6c0 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d470e640 10 monclient(hunting): authenticate will time out at 1366.736355s 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d1ca9640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f23cc0af6c0 0x7f23cc152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:48318/0 (socket says 172.21.15.107:48318) 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d1ca9640 1 -- 172.21.15.107:0/2997986471 learned_addr learned my addr 172.21.15.107:0/2997986471 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:15.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d24aa640 1 --1- 172.21.15.107:0/2997986471 >> v1:172.21.15.129:6789/0 conn(0x7f23cc0ad6c0 0x7f23cc14f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:44590/0 (socket says 172.21.15.107:44590) 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125228745 0 0) 0x7f23cc0afb60 con 0x7f23cc0af6c0 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 10 monclient(hunting): _negotiate 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b8005730 con 0x7f23cc0af6c0 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773097139 0 0) 0x7f23cc0af3a0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 10 monclient(hunting): _negotiate 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.468+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23cc0afb60 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.469+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 638067300 0 0) 0x7f23b8005730 con 0x7f23cc0af6c0 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.469+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 >> v1:172.21.15.129:6789/0 conn(0x7f23cc0ad6c0 legacy=0x7f23cc14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:15.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.469+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 >> v1:172.21.15.107:6789/0 conn(0x7f23cc14fdc0 legacy=0x7f23cc15a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:07:15.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 10 monclient: _finish_hunting 0 2024-03-29T11:07:15.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 1 monclient: found mon.noname-b 2024-03-29T11:07:15.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc0aab80 con 0x7f23cc0af6c0 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 10 monclient: _finish_auth 0 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 10 monclient: _check_auth_tickets 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f23bc0012f0 con 0x7f23cc0af6c0 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.471+0000 7f23d14a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d14a8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d14a8640 10 monclient: dump: 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.455 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d470e640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d470e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23bc0014d0 con 0x7f23cc0af6c0 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d14a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d14a8640 1 -- 172.21.15.107:0/2997986471 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f23bc0018a0 con 0x7f23cc0af6c0 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d14a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d14a8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:15.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d14a8640 10 monclient: dump: 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d470e640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.472+0000 7f23d470e640 4 set_mon_vals no callback set 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.473+0000 7f23d470e640 10 monclient: shutdown 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.473+0000 7f23d470e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:15.457 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.473+0000 7f23d470e640 1 -- 172.21.15.107:0/2997986471 >> v1:172.21.15.107:6790/0 conn(0x7f23cc0af6c0 legacy=0x7f23cc152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.473+0000 7f23d470e640 1 -- 172.21.15.107:0/2997986471 shutdown_connections 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.473+0000 7f23d470e640 1 -- 172.21.15.107:0/2997986471 >> 172.21.15.107:0/2997986471 conn(0x7f23cc0b35e0 msgr2=0x7f23cc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.474+0000 7f23d470e640 1 -- 172.21.15.107:0/2997986471 shutdown_connections 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.474+0000 7f23d470e640 1 -- 172.21.15.107:0/2997986471 wait complete. 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.475+0000 7f23d470e640 10 monclient: build_initial_monmap 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.475+0000 7f23d470e640 10 monclient: monmap: 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:07:15.458 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:15.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.475+0000 7f23d470e640 1 librados: starting msgr at 2024-03-29T11:07:15.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.475+0000 7f23d470e640 1 librados: starting objecter 2024-03-29T11:07:15.459 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.476+0000 7f23d470e640 1 Processor -- start 2024-03-29T11:07:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.476+0000 7f23d470e640 1 -- start start 2024-03-29T11:07:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.476+0000 7f23d470e640 1 librados: setting wanted keys 2024-03-29T11:07:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.476+0000 7f23d470e640 1 librados: calling monclient init 2024-03-29T11:07:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.476+0000 7f23d470e640 10 monclient: init 2024-03-29T11:07:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:07:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 10 monclient(hunting): picked mon.noname-c con 0x7f23cc0ad6c0 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 10 monclient(hunting): picked mon.noname-a con 0x7f23cc14fdc0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:15.464 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 10 monclient(hunting): picked mon.noname-b con 0x7f23cc1ed180 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0851c0 con 0x7f23cc14fdc0 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0856e0 con 0x7f23cc1ed180 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc085870 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d470e640 10 monclient(hunting): authenticate will time out at 1366.745198s 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d24aa640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f23cc0ad6c0 0x7f23cc1eca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:50052/0 (socket says 172.21.15.107:50052) 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23d24aa640 1 -- 172.21.15.107:0/192177609 learned_addr learned my addr 172.21.15.107:0/192177609 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 client.?.objecter ms_handle_connect 0x7f23cc0ad6c0 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:15.465 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 client.?.objecter ms_handle_connect 0x7f23cc1ed180 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097241540 0 0) 0x7f23cc085870 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b0004eb0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111162702 0 0) 0x7f23cc0856e0 con 0x7f23cc1ed180 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:07:15.466 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b0006230 con 0x7f23cc1ed180 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3675723961 0 0) 0x7f23b0004eb0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23cc0856e0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.477+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747190861 0 0) 0x7f23b0006230 con 0x7f23cc1ed180 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b0006ad0 con 0x7f23cc1ed180 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f23c4001a90 con 0x7f23cc1ed180 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:15.467 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:15.468 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:15.469 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:15.469 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:15.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:15.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f23c80019f0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:15.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:15.469 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:07:15.469 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:15.469 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:15.470 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.470 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.470 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:15.470 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:15.470 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:15.470 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:15.470 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:15.470 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:15.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3522931172 0 0) 0x7f23cc0856e0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 >> v1:172.21.15.129:6789/0 conn(0x7f23cc1ed180 legacy=0x7f23cc14d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:15.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.478+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 >> v1:172.21.15.107:6789/0 conn(0x7f23cc14fdc0 legacy=0x7f23cc0845d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:15.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:07:15.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23c2ffd640 1 monclient: found mon.c 2024-03-29T11:07:15.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:15.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23c2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:15.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc084ce0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.471 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:07:15.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:07:15.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:15.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23d470e640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:07:15.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23d470e640 10 monclient: _renew_subs 2024-03-29T11:07:15.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23d470e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:15.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23d470e640 1 -- 172.21.15.107:0/192177609 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc084f30 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23d470e640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:15.472 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23d470e640 10 monclient: _renew_subs 2024-03-29T11:07:15.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23d470e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:15.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23d470e640 1 -- 172.21.15.107:0/192177609 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc14df80 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.479+0000 7f23d470e640 1 librados: init done 2024-03-29T11:07:15.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23c80047c0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:15.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f23c80019f0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:15.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:15.473 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c2ffd640 10 monclient: dump: 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:15.474 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23c0ff9640 1 -- 172.21.15.107:0/192177609 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b4005080 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.481+0000 7f23d0ca7640 4 set_mon_vals no callback set 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.482+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f23c80049f0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.483+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 918919918 0 0) 0x7f23c80847b0 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.483+0000 7f23c2ffd640 10 client.4649.objecter ms_dispatch 0x7f23cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.483+0000 7f23c2ffd640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.483+0000 7f23c2ffd640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.483+0000 7f23d1ca9640 10 monclient: get_auth_request con 0x7f23b006e3d0 auth_method 0 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.483+0000 7f23c2ffd640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-29T11:07:15.475 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.483+0000 7f23c2ffd640 10 client.4649.objecter ms_handle_connect 0x7f23b006e3d0 2024-03-29T11:07:15.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.489+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23c8050090 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.489+0000 7f23c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:15.476 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.489+0000 7f23c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:15.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.792+0000 7f23c0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:15.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.792+0000 7f23c0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:15.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.792+0000 7f23c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:15.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.792+0000 7f23c0ff9640 1 -- 172.21.15.107:0/192177609 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23b4005330 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.772 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.793+0000 7f23c2ffd640 1 -- 172.21.15.107:0/192177609 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3225893291) 0x7f23c8050090 con 0x7f23cc0ad6c0 2024-03-29T11:07:15.772 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.793+0000 7f23c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:15.772 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.793+0000 7f23c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:15.772 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:07:15.773 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T11:07:15.551396+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:07:15.551396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:15.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.796+0000 7f23c0ff9640 10 librados: watch_flush enter 2024-03-29T11:07:15.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.796+0000 7f23c0ff9640 10 librados: watch_flush exit 2024-03-29T11:07:15.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.796+0000 7f23c0ff9640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-29T11:07:15.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.796+0000 7f23c0ff9640 10 client.4649.objecter successfully canceled tick 2024-03-29T11:07:15.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.796+0000 7f23c0ff9640 1 -- 172.21.15.107:0/192177609 >> v1:172.21.15.107:6816/882194755 conn(0x7f23b006e3d0 legacy=0x7f23b0070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:15.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.796+0000 7f23c0ff9640 10 monclient: shutdown 2024-03-29T11:07:15.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.796+0000 7f23c0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:15.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.796+0000 7f23c0ff9640 1 -- 172.21.15.107:0/192177609 >> v1:172.21.15.107:6790/0 conn(0x7f23cc0ad6c0 legacy=0x7f23cc1eca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:15.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.797+0000 7f23c0ff9640 1 -- 172.21.15.107:0/192177609 shutdown_connections 2024-03-29T11:07:15.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.797+0000 7f23c0ff9640 1 -- 172.21.15.107:0/192177609 >> 172.21.15.107:0/192177609 conn(0x7f23cc0b35e0 msgr2=0x7f23cc0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:15.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.797+0000 7f23c0ff9640 1 -- 172.21.15.107:0/192177609 shutdown_connections 2024-03-29T11:07:15.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.797+0000 7f23c0ff9640 1 -- 172.21.15.107:0/192177609 wait complete. 2024-03-29T11:07:15.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:15.797+0000 7f23c0ff9640 1 librados: shutdown 2024-03-29T11:07:15.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:22.598 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:22.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.763+0000 7f1318cd9640 10 monclient: get_monmap_and_config 2024-03-29T11:07:22.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.763+0000 7f1318cd9640 10 monclient: build_initial_monmap 2024-03-29T11:07:22.743 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.763+0000 7f1318cd9640 10 monclient: monmap: 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:07:22.764602+0000 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:07:22.764602+0000 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:07:22.744 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:22.745 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.765+0000 7f1318cd9640 1 Processor -- start 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.766+0000 7f1318cd9640 1 -- start start 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.766+0000 7f1318cd9640 10 monclient: init 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f13140a9ac0 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f131413ee90 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f13140af900 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13140afd10 con 0x7f13140af900 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13140af6f0 con 0x7f13140a9ac0 2024-03-29T11:07:22.750 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13140aac80 con 0x7f131413ee90 2024-03-29T11:07:22.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:22.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.767+0000 7f1318cd9640 10 monclient(hunting): authenticate will time out at 1374.035184s 2024-03-29T11:07:22.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.770+0000 7f1312d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f13140af900 0x7f1314149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37877/0 (socket says 172.21.15.107:37877) 2024-03-29T11:07:22.751 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.770+0000 7f1312d9d640 1 -- 172.21.15.107:0/553440729 learned_addr learned my addr 172.21.15.107:0/553440729 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:22.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.771+0000 7f131159a640 1 -- 172.21.15.107:0/553440729 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074646171 0 0) 0x7f13140afd10 con 0x7f13140af900 2024-03-29T11:07:22.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.771+0000 7f131159a640 10 monclient(hunting): _negotiate 2024-03-29T11:07:22.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.771+0000 7f131159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:22.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.771+0000 7f131159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:22.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.771+0000 7f131159a640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:07:22.755 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.771+0000 7f131159a640 1 -- 172.21.15.107:0/553440729 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12fc003520 con 0x7f13140af900 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 1 -- 172.21.15.107:0/553440729 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348335265 0 0) 0x7f12fc003520 con 0x7f13140af900 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 1 -- 172.21.15.107:0/553440729 >> v1:172.21.15.107:6790/0 conn(0x7f131413ee90 legacy=0x7f131413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 1 -- 172.21.15.107:0/553440729 >> v1:172.21.15.129:6789/0 conn(0x7f13140a9ac0 legacy=0x7f13140adfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 10 monclient: _finish_hunting 0 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 1 monclient: found mon.noname-a 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 1 -- 172.21.15.107:0/553440729 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13140aaf60 con 0x7f13140af900 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 10 monclient: _finish_auth 0 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 10 monclient: _check_auth_tickets 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 1 -- 172.21.15.107:0/553440729 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1304001690 con 0x7f13140af900 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:22.756 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f131159a640 10 monclient: dump: 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f1318cd9640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.773+0000 7f1318cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.776+0000 7f131159a640 1 -- 172.21.15.107:0/553440729 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1304001860 con 0x7f13140af900 2024-03-29T11:07:22.757 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.776+0000 7f131159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.776+0000 7f131159a640 1 -- 172.21.15.107:0/553440729 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f1304001cb0 con 0x7f13140af900 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.776+0000 7f131159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.776+0000 7f131159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.776+0000 7f131159a640 10 monclient: dump: 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:22.758 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:22.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.776+0000 7f1318cd9640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:22.759 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.776+0000 7f1318cd9640 4 set_mon_vals no callback set 2024-03-29T11:07:22.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.777+0000 7f1318cd9640 10 monclient: shutdown 2024-03-29T11:07:22.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.777+0000 7f1318cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:22.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.777+0000 7f1318cd9640 1 -- 172.21.15.107:0/553440729 >> v1:172.21.15.107:6789/0 conn(0x7f13140af900 legacy=0x7f1314149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:22.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.779+0000 7f1318cd9640 1 -- 172.21.15.107:0/553440729 shutdown_connections 2024-03-29T11:07:22.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.779+0000 7f1318cd9640 1 -- 172.21.15.107:0/553440729 >> 172.21.15.107:0/553440729 conn(0x7f13140b35e0 msgr2=0x7f13140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:22.760 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.779+0000 7f1318cd9640 1 -- 172.21.15.107:0/553440729 shutdown_connections 2024-03-29T11:07:22.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.780+0000 7f1318cd9640 1 -- 172.21.15.107:0/553440729 wait complete. 2024-03-29T11:07:22.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.781+0000 7f1318cd9640 10 monclient: build_initial_monmap 2024-03-29T11:07:22.761 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.781+0000 7f1318cd9640 10 monclient: monmap: 2024-03-29T11:07:22.761 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:22.761 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:22.761 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:07:22.761 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:07:22.761 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:22.762 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:22.762 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:22.762 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:07:22.762 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:07:22.762 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:22.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.781+0000 7f1318cd9640 1 librados: starting msgr at 2024-03-29T11:07:22.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.781+0000 7f1318cd9640 1 librados: starting objecter 2024-03-29T11:07:22.762 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.782+0000 7f1318cd9640 1 Processor -- start 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.782+0000 7f1318cd9640 1 -- start start 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.782+0000 7f1318cd9640 1 librados: setting wanted keys 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.782+0000 7f1318cd9640 1 librados: calling monclient init 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.782+0000 7f1318cd9640 10 monclient: init 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f13140a9ac0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f13140af900 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f131413ee90 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:22.763 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13140b2cb0 con 0x7f13140a9ac0 2024-03-29T11:07:22.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131412b1f0 con 0x7f131413ee90 2024-03-29T11:07:22.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131413cf20 con 0x7f13140af900 2024-03-29T11:07:22.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:22.764 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.783+0000 7f1318cd9640 10 monclient(hunting): authenticate will time out at 1374.051243s 2024-03-29T11:07:22.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.786+0000 7f131259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f13140a9ac0 0x7f131413d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39011/0 (socket says 172.21.15.107:39011) 2024-03-29T11:07:22.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.786+0000 7f131259c640 1 -- 172.21.15.107:0/3808400246 learned_addr learned my addr 172.21.15.107:0/3808400246 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:22.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.786+0000 7f1302ffd640 10 client.?.objecter ms_handle_connect 0x7f13140a9ac0 2024-03-29T11:07:22.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.786+0000 7f1302ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:22.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.786+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568872317 0 0) 0x7f13140b2cb0 con 0x7f13140a9ac0 2024-03-29T11:07:22.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.786+0000 7f1302ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:22.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.786+0000 7f1302ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:22.770 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.787+0000 7f1302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:22.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.787+0000 7f1302ffd640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:07:22.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.787+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f4004eb0 con 0x7f13140a9ac0 2024-03-29T11:07:22.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.787+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1609018499 0 0) 0x7f12f4004eb0 con 0x7f13140a9ac0 2024-03-29T11:07:22.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.787+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13140b2cb0 con 0x7f13140a9ac0 2024-03-29T11:07:22.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.787+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f130c001bf0 con 0x7f13140a9ac0 2024-03-29T11:07:22.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.788+0000 7f1302ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:22.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.788+0000 7f1302ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:07:22.771 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.788+0000 7f1302ffd640 10 monclient(hunting): dump: 2024-03-29T11:07:22.771 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.788+0000 7f1302ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.788+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1349084986 0 0) 0x7f13140b2cb0 con 0x7f13140a9ac0 2024-03-29T11:07:22.772 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.788+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 >> v1:172.21.15.107:6790/0 conn(0x7f13140af900 legacy=0x7f131412a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.788+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 >> v1:172.21.15.129:6789/0 conn(0x7f131413ee90 legacy=0x7f131412aae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1302ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1302ffd640 1 monclient: found mon.a 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1302ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1302ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131413d200 con 0x7f13140a9ac0 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1302ffd640 10 monclient: _finish_auth 0 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1302ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1302ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1318cd9640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1318cd9640 10 monclient: _renew_subs 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1318cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131413c0c0 con 0x7f13140a9ac0 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1318cd9640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:22.773 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1318cd9640 10 monclient: _renew_subs 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1318cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13140b3320 con 0x7f13140a9ac0 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.789+0000 7f1318cd9640 1 librados: init done 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.790+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f130c001d50 con 0x7f13140a9ac0 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.790+0000 7f1302ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.790+0000 7f1310d99640 4 set_mon_vals no callback set 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.791+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f130c004e60 con 0x7f13140a9ac0 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.791+0000 7f1302ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.791+0000 7f1302ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.791+0000 7f1302ffd640 10 monclient: dump: 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:22.774 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:22.775 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:22.775 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:22.775 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:22.775 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:22.775 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:22.775 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.792+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f130c004fc0 con 0x7f13140a9ac0 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2565443581 0 0) 0x7f130c084ee0 con 0x7f13140a9ac0 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1302ffd640 10 client.4560.objecter ms_dispatch 0x7f1314000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1302ffd640 3 client.4560.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1311d9b640 10 monclient: get_auth_request con 0x7f12f406dc80 auth_method 0 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1318cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1318cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1318cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1302ffd640 3 client.4560.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:07:22.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1302ffd640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T11:07:22.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f8005080 con 0x7f13140a9ac0 2024-03-29T11:07:22.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.793+0000 7f1302ffd640 10 client.4560.objecter ms_handle_connect 0x7f12f406dc80 2024-03-29T11:07:22.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.802+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f130c0505f0 con 0x7f13140a9ac0 2024-03-29T11:07:22.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.802+0000 7f1302ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:22.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:22.802+0000 7f1302ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:23.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.127+0000 7f1318cd9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:23.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.127+0000 7f1318cd9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:23.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.127+0000 7f1318cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:07:23.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.127+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12f8005330 con 0x7f13140a9ac0 2024-03-29T11:07:23.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.128+0000 7f1302ffd640 1 -- 172.21.15.107:0/3808400246 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3225893291) 0x7f130c0526f0 con 0x7f13140a9ac0 2024-03-29T11:07:23.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.128+0000 7f1302ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:23.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.128+0000 7f1302ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:23.108 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:07:23.108 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T11:07:15.551396+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:07:15.551396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:23.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.131+0000 7f1318cd9640 10 librados: watch_flush enter 2024-03-29T11:07:23.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.131+0000 7f1318cd9640 10 librados: watch_flush exit 2024-03-29T11:07:23.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.131+0000 7f1318cd9640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T11:07:23.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.131+0000 7f1318cd9640 10 client.4560.objecter successfully canceled tick 2024-03-29T11:07:23.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.133+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 >> v1:172.21.15.107:6816/882194755 conn(0x7f12f406dc80 legacy=0x7f12f4070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:23.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.133+0000 7f1318cd9640 10 monclient: shutdown 2024-03-29T11:07:23.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.133+0000 7f1318cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:23.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.133+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 >> v1:172.21.15.107:6789/0 conn(0x7f13140a9ac0 legacy=0x7f131413d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:23.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.133+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 shutdown_connections 2024-03-29T11:07:23.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.133+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 >> 172.21.15.107:0/3808400246 conn(0x7f13140b35e0 msgr2=0x7f1314132ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:23.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.133+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 shutdown_connections 2024-03-29T11:07:23.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.133+0000 7f1318cd9640 1 -- 172.21.15.107:0/3808400246 wait complete. 2024-03-29T11:07:23.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:23.134+0000 7f1318cd9640 1 librados: shutdown 2024-03-29T11:07:23.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:29.935 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:30.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.104+0000 7f44dd40d640 10 monclient: get_monmap_and_config 2024-03-29T11:07:30.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.104+0000 7f44dd40d640 10 monclient: build_initial_monmap 2024-03-29T11:07:30.083 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.104+0000 7f44dd40d640 10 monclient: monmap: 2024-03-29T11:07:30.083 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:30.084 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:30.084 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:07:30.105351+0000 2024-03-29T11:07:30.084 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:07:30.105351+0000 2024-03-29T11:07:30.084 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:30.084 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:30.084 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:30.084 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:07:30.084 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:07:30.084 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:30.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.105+0000 7f44dd40d640 1 Processor -- start 2024-03-29T11:07:30.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.105+0000 7f44dd40d640 1 -- start start 2024-03-29T11:07:30.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.105+0000 7f44dd40d640 10 monclient: init 2024-03-29T11:07:30.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.106+0000 7f44dd40d640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:30.085 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.106+0000 7f44dd40d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:07:30.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.107+0000 7f44dd40d640 10 monclient(hunting): picked mon.noname-c con 0x7f44d80b0860 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:30.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.107+0000 7f44dd40d640 10 monclient(hunting): picked mon.noname-a con 0x7f44d813ef30 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:30.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.107+0000 7f44dd40d640 10 monclient(hunting): picked mon.noname-b con 0x7f44d806d9d0 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:30.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.107+0000 7f44dd40d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d80b0c40 con 0x7f44d813ef30 2024-03-29T11:07:30.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.107+0000 7f44dd40d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d80aed10 con 0x7f44d80b0860 2024-03-29T11:07:30.087 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.108+0000 7f44dd40d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d80ab1b0 con 0x7f44d806d9d0 2024-03-29T11:07:30.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.108+0000 7f44dd40d640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:30.088 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.108+0000 7f44dd40d640 10 monclient(hunting): authenticate will time out at 1381.375994s 2024-03-29T11:07:30.089 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.111+0000 7f44d77fe640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f44d806d9d0 0x7f44d814a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:47264/0 (socket says 172.21.15.107:47264) 2024-03-29T11:07:30.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.111+0000 7f44d77fe640 1 -- 172.21.15.107:0/341590566 learned_addr learned my addr 172.21.15.107:0/341590566 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:30.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.111+0000 7f44d67fc640 1 -- 172.21.15.107:0/341590566 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865731750 0 0) 0x7f44d80ab1b0 con 0x7f44d806d9d0 2024-03-29T11:07:30.090 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:07:30.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:30.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:30.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:07:30.091 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 1 -- 172.21.15.107:0/341590566 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c4003520 con 0x7f44d806d9d0 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 1 -- 172.21.15.107:0/341590566 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4291672209 0 0) 0x7f44c4003520 con 0x7f44d806d9d0 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 1 -- 172.21.15.107:0/341590566 >> v1:172.21.15.129:6789/0 conn(0x7f44d80b0860 legacy=0x7f44d813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 1 -- 172.21.15.107:0/341590566 >> v1:172.21.15.107:6789/0 conn(0x7f44d813ef30 legacy=0x7f44d80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 1 monclient: found mon.noname-b 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 1 -- 172.21.15.107:0/341590566 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d80ab490 con 0x7f44d806d9d0 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient: _finish_auth 0 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 1 -- 172.21.15.107:0/341590566 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f44d0001a10 con 0x7f44d806d9d0 2024-03-29T11:07:30.093 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient: dump: 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:30.094 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44dd40d640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44dd40d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 1 -- 172.21.15.107:0/341590566 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44d0001be0 con 0x7f44d806d9d0 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44d67fc640 1 -- 172.21.15.107:0/341590566 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f44d0002030 con 0x7f44d806d9d0 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.112+0000 7f44dd40d640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.113+0000 7f44d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.113+0000 7f44d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.113+0000 7f44d67fc640 10 monclient: dump: 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:30.095 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:30.096 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:30.096 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:30.096 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:30.096 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.113+0000 7f44dd40d640 4 set_mon_vals no callback set 2024-03-29T11:07:30.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.115+0000 7f44dd40d640 10 monclient: shutdown 2024-03-29T11:07:30.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.115+0000 7f44dd40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:30.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.115+0000 7f44dd40d640 1 -- 172.21.15.107:0/341590566 >> v1:172.21.15.107:6790/0 conn(0x7f44d806d9d0 legacy=0x7f44d814a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:30.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.117+0000 7f44dd40d640 1 -- 172.21.15.107:0/341590566 shutdown_connections 2024-03-29T11:07:30.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.117+0000 7f44dd40d640 1 -- 172.21.15.107:0/341590566 >> 172.21.15.107:0/341590566 conn(0x7f44d806b660 msgr2=0x7f44d806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:30.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.117+0000 7f44dd40d640 1 -- 172.21.15.107:0/341590566 shutdown_connections 2024-03-29T11:07:30.097 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.117+0000 7f44dd40d640 1 -- 172.21.15.107:0/341590566 wait complete. 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.119+0000 7f44dd40d640 10 monclient: build_initial_monmap 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.119+0000 7f44dd40d640 10 monclient: monmap: 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:07:30.098 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:30.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.119+0000 7f44dd40d640 1 librados: starting msgr at 2024-03-29T11:07:30.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.119+0000 7f44dd40d640 1 librados: starting objecter 2024-03-29T11:07:30.101 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 1 Processor -- start 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 1 -- start start 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 1 librados: setting wanted keys 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 1 librados: calling monclient init 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 10 monclient: init 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 10 monclient(hunting): picked mon.noname-a con 0x7f44d806d9d0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 10 monclient(hunting): picked mon.noname-c con 0x7f44d80b0860 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 10 monclient(hunting): picked mon.noname-b con 0x7f44d813ef30 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d806e8c0 con 0x7f44d806d9d0 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d806ff40 con 0x7f44d813ef30 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d8070220 con 0x7f44d80b0860 2024-03-29T11:07:30.102 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.120+0000 7f44dd40d640 10 monclient(hunting): authenticate will time out at 1381.388968s 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.121+0000 7f44cffff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f44d80b0860 0x7f44d8135e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:54334/0 (socket says 172.21.15.107:54334) 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.121+0000 7f44cffff640 1 -- 172.21.15.107:0/1095309163 learned_addr learned my addr 172.21.15.107:0/1095309163 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.121+0000 7f44cf7fe640 10 client.?.objecter ms_handle_connect 0x7f44d80b0860 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.121+0000 7f44cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.121+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721035613 0 0) 0x7f44d8070220 con 0x7f44d80b0860 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.121+0000 7f44cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.121+0000 7f44cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.121+0000 7f44cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.121+0000 7f44cf7fe640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:07:30.103 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.122+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b8004eb0 con 0x7f44d80b0860 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.123+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1517060718 0 0) 0x7f44b8004eb0 con 0x7f44d80b0860 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.123+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44d8070220 con 0x7f44d80b0860 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.123+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f44c0001970 con 0x7f44d80b0860 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.123+0000 7f44cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.123+0000 7f44cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.123+0000 7f44cf7fe640 10 monclient(hunting): dump: 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:30.105 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.123+0000 7f44cf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.124+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1451870965 0 0) 0x7f44d8070220 con 0x7f44d80b0860 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.124+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 >> v1:172.21.15.129:6789/0 conn(0x7f44d813ef30 legacy=0x7f44d81dfd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 >> v1:172.21.15.107:6789/0 conn(0x7f44d806d9d0 legacy=0x7f44d80709b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 1 monclient: found mon.c 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:30.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d8138840 con 0x7f44d80b0860 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44dd40d640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44dd40d640 10 monclient: _renew_subs 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44dd40d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d806e100 con 0x7f44d80b0860 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44dd40d640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44dd40d640 10 monclient: _renew_subs 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44dd40d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d806f150 con 0x7f44d80b0860 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44dd40d640 1 librados: init done 2024-03-29T11:07:30.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44c0001ad0 con 0x7f44d80b0860 2024-03-29T11:07:30.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:30.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f44c0003a60 con 0x7f44d80b0860 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44cf7fe640 10 monclient: dump: 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:30.110 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.125+0000 7f44d57fa640 4 set_mon_vals no callback set 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.126+0000 7f44dd40d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.126+0000 7f44dd40d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.126+0000 7f44dd40d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.126+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44bc005080 con 0x7f44d80b0860 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.129+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f44c0003bc0 con 0x7f44d80b0860 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.129+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2565443581 0 0) 0x7f44c0084b50 con 0x7f44d80b0860 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.129+0000 7f44cf7fe640 10 client.4655.objecter ms_dispatch 0x7f44d8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.129+0000 7f44cf7fe640 3 client.4655.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.129+0000 7f44cf7fe640 3 client.4655.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.129+0000 7f44d6ffd640 10 monclient: get_auth_request con 0x7f44b806dd00 auth_method 0 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.130+0000 7f44cf7fe640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-29T11:07:30.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.130+0000 7f44cf7fe640 10 client.4655.objecter ms_handle_connect 0x7f44b806dd00 2024-03-29T11:07:30.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.136+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44c0050260 con 0x7f44d80b0860 2024-03-29T11:07:30.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.136+0000 7f44cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:30.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.136+0000 7f44cf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:30.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.460+0000 7f44dd40d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:30.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.460+0000 7f44dd40d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:30.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.460+0000 7f44dd40d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:30.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.460+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44bc005330 con 0x7f44d80b0860 2024-03-29T11:07:30.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.461+0000 7f44cf7fe640 1 -- 172.21.15.107:0/1095309163 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3225893291) 0x7f44c0052360 con 0x7f44d80b0860 2024-03-29T11:07:30.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.461+0000 7f44cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:30.440 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.461+0000 7f44cf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:30.442 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:07:30.442 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T11:07:15.551396+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:07:15.551396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:30.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 10 librados: watch_flush enter 2024-03-29T11:07:30.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 10 librados: watch_flush exit 2024-03-29T11:07:30.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-29T11:07:30.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 10 client.4655.objecter successfully canceled tick 2024-03-29T11:07:30.445 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 >> v1:172.21.15.107:6816/882194755 conn(0x7f44b806dd00 legacy=0x7f44b80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:30.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 10 monclient: shutdown 2024-03-29T11:07:30.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:30.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 >> v1:172.21.15.107:6790/0 conn(0x7f44d80b0860 legacy=0x7f44d8135e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:30.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 shutdown_connections 2024-03-29T11:07:30.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 >> 172.21.15.107:0/1095309163 conn(0x7f44d806b660 msgr2=0x7f44d813dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:30.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 shutdown_connections 2024-03-29T11:07:30.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.466+0000 7f44dd40d640 1 -- 172.21.15.107:0/1095309163 wait complete. 2024-03-29T11:07:30.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:30.467+0000 7f44dd40d640 1 librados: shutdown 2024-03-29T11:07:30.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:37.270 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:37.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.446+0000 7fa7954eb640 10 monclient: get_monmap_and_config 2024-03-29T11:07:37.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.446+0000 7fa7954eb640 10 monclient: build_initial_monmap 2024-03-29T11:07:37.426 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.446+0000 7fa7954eb640 10 monclient: monmap: 2024-03-29T11:07:37.426 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:37.426 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:37.426 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:07:37.447359+0000 2024-03-29T11:07:37.426 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:07:37.447359+0000 2024-03-29T11:07:37.426 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:37.427 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:37.427 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:37.427 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:07:37.427 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:07:37.427 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:37.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.447+0000 7fa7954eb640 1 Processor -- start 2024-03-29T11:07:37.428 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.447+0000 7fa7954eb640 1 -- start start 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.447+0000 7fa7954eb640 10 monclient: init 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.448+0000 7fa7954eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.449+0000 7fa7954eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.449+0000 7fa7954eb640 10 monclient(hunting): picked mon.noname-b con 0x7fa788006410 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.449+0000 7fa7954eb640 10 monclient(hunting): picked mon.noname-a con 0x7fa78802a640 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.449+0000 7fa7954eb640 10 monclient(hunting): picked mon.noname-c con 0x7fa7880327d0 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.449+0000 7fa7954eb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78802c820 con 0x7fa78802a640 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.449+0000 7fa7954eb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78802a120 con 0x7fa7880327d0 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.449+0000 7fa7954eb640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7880c75b0 con 0x7fa788006410 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.449+0000 7fa7954eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:37.429 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.449+0000 7fa7954eb640 10 monclient(hunting): authenticate will time out at 1388.717129s 2024-03-29T11:07:37.431 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.453+0000 7fa78f7fe640 1 --1- >> v1:172.21.15.129:6789/0 conn(0x7fa7880327d0 0x7fa7880d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.129:6789/0 says I am v1:172.21.15.107:55457/0 (socket says 172.21.15.107:55457) 2024-03-29T11:07:37.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.453+0000 7fa78f7fe640 1 -- 172.21.15.107:0/3616031880 learned_addr learned my addr 172.21.15.107:0/3616031880 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:37.432 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.452+0000 7fa78effd640 1 --1- 172.21.15.107:0/3616031880 >> v1:172.21.15.107:6790/0 conn(0x7fa788006410 0x7fa7880c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:60664/0 (socket says 172.21.15.107:60664) 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251216509 0 0) 0x7fa78802a120 con 0x7fa7880327d0 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa778003520 con 0x7fa7880327d0 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211376917 0 0) 0x7fa7880c75b0 con 0x7fa788006410 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:37.433 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:37.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:07:37.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa78802a120 con 0x7fa788006410 2024-03-29T11:07:37.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 767199923 0 0) 0x7fa78802a120 con 0x7fa788006410 2024-03-29T11:07:37.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 >> v1:172.21.15.129:6789/0 conn(0x7fa7880327d0 legacy=0x7fa7880d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:37.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 >> v1:172.21.15.107:6789/0 conn(0x7fa78802a640 legacy=0x7fa7880320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:37.434 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 1 monclient: found mon.noname-b 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788033110 con 0x7fa788006410 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.454+0000 7fa78dffb640 10 monclient: _finish_auth 0 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa784001670 con 0x7fa788006410 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 10 monclient: dump: 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.435 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa784001890 con 0x7fa788006410 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 1 -- 172.21.15.107:0/3616031880 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa784001ba0 con 0x7fa788006410 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa78dffb640 10 monclient: dump: 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:37.436 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa7954eb640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa7954eb640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:37.437 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.455+0000 7fa7954eb640 4 set_mon_vals no callback set 2024-03-29T11:07:37.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.456+0000 7fa7954eb640 10 monclient: shutdown 2024-03-29T11:07:37.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.456+0000 7fa7954eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:37.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.456+0000 7fa7954eb640 1 -- 172.21.15.107:0/3616031880 >> v1:172.21.15.107:6790/0 conn(0x7fa788006410 legacy=0x7fa7880c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:37.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.457+0000 7fa7954eb640 1 -- 172.21.15.107:0/3616031880 shutdown_connections 2024-03-29T11:07:37.438 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.457+0000 7fa7954eb640 1 -- 172.21.15.107:0/3616031880 >> 172.21.15.107:0/3616031880 conn(0x7fa7880058f0 msgr2=0x7fa788005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:37.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.458+0000 7fa7954eb640 1 -- 172.21.15.107:0/3616031880 shutdown_connections 2024-03-29T11:07:37.439 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.458+0000 7fa7954eb640 1 -- 172.21.15.107:0/3616031880 wait complete. 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.459+0000 7fa7954eb640 10 monclient: build_initial_monmap 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.459+0000 7fa7954eb640 10 monclient: monmap: 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:07:37.441 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.459+0000 7fa7954eb640 1 librados: starting msgr at 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.459+0000 7fa7954eb640 1 librados: starting objecter 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.460+0000 7fa7954eb640 1 Processor -- start 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.460+0000 7fa7954eb640 1 -- start start 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.460+0000 7fa7954eb640 1 librados: setting wanted keys 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.460+0000 7fa7954eb640 1 librados: calling monclient init 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.460+0000 7fa7954eb640 10 monclient: init 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.460+0000 7fa7954eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.460+0000 7fa7954eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.461+0000 7fa7954eb640 10 monclient(hunting): picked mon.noname-c con 0x7fa788006410 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.461+0000 7fa7954eb640 10 monclient(hunting): picked mon.noname-b con 0x7fa78802a640 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.461+0000 7fa7954eb640 10 monclient(hunting): picked mon.noname-a con 0x7fa7880327d0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.461+0000 7fa7954eb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7880c39a0 con 0x7fa7880327d0 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.461+0000 7fa7954eb640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7880c45e0 con 0x7fa78802a640 2024-03-29T11:07:37.442 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.461+0000 7fa7954eb640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7880c48c0 con 0x7fa788006410 2024-03-29T11:07:37.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.461+0000 7fa7954eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:37.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.461+0000 7fa7954eb640 10 monclient(hunting): authenticate will time out at 1388.729048s 2024-03-29T11:07:37.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.463+0000 7fa78effd640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fa788006410 0x7fa7880c27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:33310/0 (socket says 172.21.15.107:33310) 2024-03-29T11:07:37.443 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.463+0000 7fa78effd640 1 -- 172.21.15.107:0/1524861902 learned_addr learned my addr 172.21.15.107:0/1524861902 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:37.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.467+0000 7fa7777fe640 10 client.?.objecter ms_handle_connect 0x7fa788006410 2024-03-29T11:07:37.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.467+0000 7fa7777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:37.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.467+0000 7fa7777fe640 10 client.?.objecter ms_handle_connect 0x7fa78802a640 2024-03-29T11:07:37.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.467+0000 7fa7777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:37.446 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 client.?.objecter ms_handle_connect 0x7fa7880327d0 2024-03-29T11:07:37.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:37.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892262428 0 0) 0x7fa7880c45e0 con 0x7fa78802a640 2024-03-29T11:07:37.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:37.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:37.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:37.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:07:37.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa77c004eb0 con 0x7fa78802a640 2024-03-29T11:07:37.447 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045594973 0 0) 0x7fa7880c39a0 con 0x7fa7880327d0 2024-03-29T11:07:37.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:37.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:37.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:37.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:07:37.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.468+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7880c45e0 con 0x7fa7880327d0 2024-03-29T11:07:37.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668572447 0 0) 0x7fa7880c48c0 con 0x7fa788006410 2024-03-29T11:07:37.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:37.448 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7880c39a0 con 0x7fa788006410 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 580027471 0 0) 0x7fa77c004eb0 con 0x7fa78802a640 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.129:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7880c48c0 con 0x7fa78802a640 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa79004fdd0 con 0x7fa78802a640 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): dump: 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.449 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2649971503 0 0) 0x7fa7880c45e0 con 0x7fa7880327d0 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa77c004eb0 con 0x7fa7880327d0 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa780001880 con 0x7fa7880327d0 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient(hunting): dump: 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:37.450 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2212733504 0 0) 0x7fa7880c48c0 con 0x7fa78802a640 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 >> v1:172.21.15.107:6790/0 conn(0x7fa788006410 legacy=0x7fa7880c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 >> v1:172.21.15.107:6789/0 conn(0x7fa7880327d0 legacy=0x7fa7880bf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.469+0000 7fa7777fe640 1 monclient: found mon.b 2024-03-29T11:07:37.451 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.470+0000 7fa7777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:37.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.470+0000 7fa7777fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:37.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.470+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7880c3480 con 0x7fa78802a640 2024-03-29T11:07:37.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.470+0000 7fa7777fe640 10 monclient: _finish_auth 0 2024-03-29T11:07:37.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.470+0000 7fa7777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:07:37.452 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.470+0000 7fa7777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.470+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa79004f400 con 0x7fa78802a640 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.470+0000 7fa7777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.470+0000 7fa78d7fa640 4 set_mon_vals no callback set 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fa790075f10 con 0x7fa78802a640 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7777fe640 10 monclient: dump: 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:37.453 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7954eb640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7954eb640 10 monclient: _renew_subs 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7954eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.129:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7880d8a30 con 0x7fa78802a640 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7954eb640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7954eb640 10 monclient: _renew_subs 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7954eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.129:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7880d8ec0 con 0x7fa78802a640 2024-03-29T11:07:37.454 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.471+0000 7fa7954eb640 1 librados: init done 2024-03-29T11:07:37.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.474+0000 7fa7954eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:37.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.474+0000 7fa7954eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:37.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.474+0000 7fa7954eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:37.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.474+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa788004cd0 con 0x7fa78802a640 2024-03-29T11:07:37.455 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.474+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fa790067bf0 con 0x7fa78802a640 2024-03-29T11:07:37.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.475+0000 7fa78effd640 10 monclient: get_auth_request con 0x7fa77c06de00 auth_method 0 2024-03-29T11:07:37.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.476+0000 7fa7777fe640 10 client.4528.objecter ms_handle_connect 0x7fa77c06de00 2024-03-29T11:07:37.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.476+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2565443581 0 0) 0x7fa7900fe230 con 0x7fa78802a640 2024-03-29T11:07:37.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.476+0000 7fa7777fe640 10 client.4528.objecter ms_dispatch 0x7fa790000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:07:37.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.476+0000 7fa7777fe640 3 client.4528.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:07:37.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.476+0000 7fa7777fe640 3 client.4528.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:07:37.456 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.476+0000 7fa7777fe640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T11:07:37.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.479+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7900c9b10 con 0x7fa78802a640 2024-03-29T11:07:37.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.480+0000 7fa7777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:37.458 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.480+0000 7fa7777fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:37.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.818+0000 7fa7954eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:37.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.818+0000 7fa7954eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:37.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.818+0000 7fa7954eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.129:6789/0 2024-03-29T11:07:37.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.818+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 --> v1:172.21.15.129:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7880d1e30 con 0x7fa78802a640 2024-03-29T11:07:37.803 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:07:37.803 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T11:07:15.551396+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:07:15.551396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:37.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.822+0000 7fa7777fe640 1 -- 172.21.15.107:0/1524861902 <== mon.1 v1:172.21.15.129:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3225893291) 0x7fa7900cba40 con 0x7fa78802a640 2024-03-29T11:07:37.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.822+0000 7fa7777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:37.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.822+0000 7fa7777fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:37.806 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.827+0000 7fa7954eb640 10 librados: watch_flush enter 2024-03-29T11:07:37.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.830+0000 7fa7954eb640 10 librados: watch_flush exit 2024-03-29T11:07:37.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.830+0000 7fa7954eb640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T11:07:37.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.830+0000 7fa7954eb640 10 client.4528.objecter successfully canceled tick 2024-03-29T11:07:37.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.830+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 >> v1:172.21.15.107:6816/882194755 conn(0x7fa77c06de00 legacy=0x7fa77c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:37.809 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.830+0000 7fa7954eb640 10 monclient: shutdown 2024-03-29T11:07:37.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.830+0000 7fa7954eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:37.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.830+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 >> v1:172.21.15.129:6789/0 conn(0x7fa78802a640 legacy=0x7fa7880d8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:37.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.830+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 shutdown_connections 2024-03-29T11:07:37.810 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.830+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 >> 172.21.15.107:0/1524861902 conn(0x7fa7880058f0 msgr2=0x7fa7880043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:37.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.832+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 shutdown_connections 2024-03-29T11:07:37.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.833+0000 7fa7954eb640 1 -- 172.21.15.107:0/1524861902 wait complete. 2024-03-29T11:07:37.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:37.833+0000 7fa7954eb640 1 librados: shutdown 2024-03-29T11:07:37.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:44.638 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:44.774 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.796+0000 7f3b8660f640 10 monclient: get_monmap_and_config 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.796+0000 7f3b8660f640 10 monclient: build_initial_monmap 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.796+0000 7f3b8660f640 10 monclient: monmap: 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:07:44.796999+0000 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:07:44.796999+0000 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:07:44.775 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:44.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.797+0000 7f3b8660f640 1 Processor -- start 2024-03-29T11:07:44.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.797+0000 7f3b8660f640 1 -- start start 2024-03-29T11:07:44.776 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.797+0000 7f3b8660f640 10 monclient: init 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 10 monclient(hunting): picked mon.noname-b con 0x7f3b801422b0 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 10 monclient(hunting): picked mon.noname-c con 0x7f3b80142690 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 10 monclient(hunting): picked mon.noname-a con 0x7f3b800b17d0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b80142c80 con 0x7f3b800b17d0 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b800aae40 con 0x7f3b80142690 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8013fbd0 con 0x7f3b801422b0 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b8660f640 10 monclient(hunting): authenticate will time out at 1396.066144s 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b84bac640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3b800b17d0 0x7f3b800b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48036/0 (socket says 172.21.15.107:48036) 2024-03-29T11:07:44.777 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.798+0000 7f3b84bac640 1 -- 172.21.15.107:0/897686157 learned_addr learned my addr 172.21.15.107:0/897686157 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020351537 0 0) 0x7f3b80142c80 con 0x7f3b800b17d0 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b68003520 con 0x7f3b800b17d0 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306176161 0 0) 0x7f3b8013fbd0 con 0x7f3b801422b0 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:44.779 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:44.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:07:44.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b80142c80 con 0x7f3b801422b0 2024-03-29T11:07:44.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3681116061 0 0) 0x7f3b68003520 con 0x7f3b800b17d0 2024-03-29T11:07:44.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 >> v1:172.21.15.107:6790/0 conn(0x7f3b801422b0 legacy=0x7f3b8013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:44.780 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.799+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 >> v1:172.21.15.129:6789/0 conn(0x7f3b80142690 legacy=0x7f3b800b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:07:44.782 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 10 monclient: _finish_hunting 0 2024-03-29T11:07:44.782 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 1 monclient: found mon.noname-a 2024-03-29T11:07:44.782 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:44.782 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.107:6789/0 2024-03-29T11:07:44.782 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b80151940 con 0x7f3b800b17d0 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 10 monclient: _finish_auth 0 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 10 monclient: _check_auth_tickets 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b8660f640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b8660f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3b78001a10 con 0x7f3b800b17d0 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 10 monclient: dump: 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:44.783 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b8660f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.801+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 <== mon.0 v1:172.21.15.107:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b78001c30 con 0x7f3b800b17d0 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b8660f640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b8660f640 4 set_mon_vals no callback set 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b7effd640 1 -- 172.21.15.107:0/897686157 <== mon.0 v1:172.21.15.107:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3b78001f40 con 0x7f3b800b17d0 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:07:44.784 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b7effd640 10 monclient: dump: 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b8660f640 10 monclient: shutdown 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b8660f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:44.785 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.802+0000 7f3b8660f640 1 -- 172.21.15.107:0/897686157 >> v1:172.21.15.107:6789/0 conn(0x7f3b800b17d0 legacy=0x7f3b800b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:44.792 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.805+0000 7f3b8660f640 1 -- 172.21.15.107:0/897686157 shutdown_connections 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.805+0000 7f3b8660f640 1 -- 172.21.15.107:0/897686157 >> 172.21.15.107:0/897686157 conn(0x7f3b80066a00 msgr2=0x7f3b80066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.805+0000 7f3b8660f640 1 -- 172.21.15.107:0/897686157 shutdown_connections 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.805+0000 7f3b8660f640 1 -- 172.21.15.107:0/897686157 wait complete. 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.806+0000 7f3b8660f640 10 monclient: build_initial_monmap 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.806+0000 7f3b8660f640 10 monclient: monmap: 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:07:44.793 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.806+0000 7f3b8660f640 1 librados: starting msgr at 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.806+0000 7f3b8660f640 1 librados: starting objecter 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.806+0000 7f3b8660f640 1 Processor -- start 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.807+0000 7f3b8660f640 1 -- start start 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.807+0000 7f3b8660f640 1 librados: setting wanted keys 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.807+0000 7f3b8660f640 1 librados: calling monclient init 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.807+0000 7f3b8660f640 10 monclient: init 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:44.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 10 monclient(hunting): picked mon.noname-b con 0x7f3b800b17d0 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 10 monclient(hunting): picked mon.noname-a con 0x7f3b801422b0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 10 monclient(hunting): picked mon.noname-c con 0x7f3b80142690 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b80150be0 con 0x7f3b801422b0 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8012f510 con 0x7f3b800b17d0 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8012f6a0 con 0x7f3b80142690 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b8660f640 10 monclient(hunting): authenticate will time out at 1396.076151s 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b84bac640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f3b80142690 0x7f3b8014f2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:38710/0 (socket says 172.21.15.107:38710) 2024-03-29T11:07:44.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b84bac640 1 -- 172.21.15.107:0/144175606 learned_addr learned my addr 172.21.15.107:0/144175606 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 client.?.objecter ms_handle_connect 0x7f3b80142690 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275571603 0 0) 0x7f3b8012f6a0 con 0x7f3b80142690 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b6c004eb0 con 0x7f3b80142690 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1989649141 0 0) 0x7f3b6c004eb0 con 0x7f3b80142690 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b8012f6a0 con 0x7f3b80142690 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 client.?.objecter ms_handle_connect 0x7f3b801422b0 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3b78001910 con 0x7f3b80142690 2024-03-29T11:07:44.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 monclient(hunting): dump: 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.808+0000 7f3b7cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.809+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 130032996 0 0) 0x7f3b8012f6a0 con 0x7f3b80142690 2024-03-29T11:07:44.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.809+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 >> v1:172.21.15.129:6789/0 conn(0x7f3b800b17d0 legacy=0x7f3b8006de90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:44.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 >> v1:172.21.15.107:6789/0 conn(0x7f3b801422b0 legacy=0x7f3b800b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b7cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b7cff9640 1 monclient: found mon.c 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b7cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b7cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8014f9b0 con 0x7f3b80142690 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b7cff9640 10 monclient: _finish_auth 0 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b7cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b8660f640 5 monclient: authenticate success, global_id 4667 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b8660f640 10 monclient: _renew_subs 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b8660f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.810+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b80151070 con 0x7f3b80142690 2024-03-29T11:07:44.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.811+0000 7f3b8660f640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.811+0000 7f3b8660f640 10 monclient: _renew_subs 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.811+0000 7f3b8660f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.811+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b801efc30 con 0x7f3b80142690 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.811+0000 7f3b8660f640 1 librados: init done 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.812+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b78001a70 con 0x7f3b80142690 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.812+0000 7f3b7cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.812+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f3b78004fb0 con 0x7f3b80142690 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.812+0000 7f3b7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.812+0000 7f3b7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.812+0000 7f3b7cff9640 10 monclient: dump: 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:44.800 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.812+0000 7f3b8660f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.812+0000 7f3b7e7fc640 4 set_mon_vals no callback set 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.813+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f3b780051d0 con 0x7f3b80142690 2024-03-29T11:07:44.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.814+0000 7f3b8660f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.814+0000 7f3b8660f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.814+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8006f280 con 0x7f3b80142690 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.815+0000 7f3b7ffff640 10 monclient: get_auth_request con 0x7f3b6c06dae0 auth_method 0 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.815+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2565443581 0 0) 0x7f3b780860c0 con 0x7f3b80142690 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.815+0000 7f3b7cff9640 10 client.4667.objecter ms_dispatch 0x7f3b80000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.815+0000 7f3b7cff9640 3 client.4667.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.815+0000 7f3b7cff9640 3 client.4667.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.816+0000 7f3b7cff9640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.816+0000 7f3b7cff9640 10 client.4667.objecter ms_handle_connect 0x7f3b6c06dae0 2024-03-29T11:07:44.803 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.822+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b78050970 con 0x7f3b80142690 2024-03-29T11:07:44.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.822+0000 7f3b7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:44.804 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:44.822+0000 7f3b7cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:45.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.148+0000 7f3b8660f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:45.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.148+0000 7f3b8660f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:45.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.148+0000 7f3b8660f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:45.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.148+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b80150d90 con 0x7f3b80142690 2024-03-29T11:07:45.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.148+0000 7f3b7cff9640 1 -- 172.21.15.107:0/144175606 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3225893291) 0x7f3b780528a0 con 0x7f3b80142690 2024-03-29T11:07:45.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.149+0000 7f3b7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:45.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.149+0000 7f3b7cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:45.128 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:07:45.128 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T11:07:15.551396+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:07:15.551396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.152+0000 7f3b8660f640 10 librados: watch_flush enter 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.152+0000 7f3b8660f640 10 librados: watch_flush exit 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.152+0000 7f3b8660f640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.152+0000 7f3b8660f640 10 client.4667.objecter successfully canceled tick 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.153+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 >> v1:172.21.15.107:6816/882194755 conn(0x7f3b6c06dae0 legacy=0x7f3b6c06ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.153+0000 7f3b8660f640 10 monclient: shutdown 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.153+0000 7f3b8660f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.153+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 >> v1:172.21.15.107:6790/0 conn(0x7f3b80142690 legacy=0x7f3b8014f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.153+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 shutdown_connections 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.153+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 >> 172.21.15.107:0/144175606 conn(0x7f3b80066a00 msgr2=0x7f3b800b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.153+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 shutdown_connections 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.153+0000 7f3b8660f640 1 -- 172.21.15.107:0/144175606 wait complete. 2024-03-29T11:07:45.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:45.153+0000 7f3b8660f640 1 librados: shutdown 2024-03-29T11:07:45.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:51.955 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:52.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.127+0000 7f630e564640 10 monclient: get_monmap_and_config 2024-03-29T11:07:52.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.127+0000 7f630e564640 10 monclient: build_initial_monmap 2024-03-29T11:07:52.106 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.127+0000 7f630e564640 10 monclient: monmap: 2024-03-29T11:07:52.106 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:52.106 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:52.107 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:07:52.128436+0000 2024-03-29T11:07:52.107 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:07:52.128436+0000 2024-03-29T11:07:52.107 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:52.107 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:52.107 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:52.107 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:07:52.107 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:07:52.107 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:52.107 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.129+0000 7f630e564640 1 Processor -- start 2024-03-29T11:07:52.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.129+0000 7f630e564640 1 -- start start 2024-03-29T11:07:52.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.129+0000 7f630e564640 10 monclient: init 2024-03-29T11:07:52.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:52.108 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:07:52.109 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 10 monclient(hunting): picked mon.noname-b con 0x7f63080708a0 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 10 monclient(hunting): picked mon.noname-c con 0x7f630806fe90 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 10 monclient(hunting): picked mon.noname-a con 0x7f6308143c50 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f6307fff640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f63080708a0 0x7f63080af390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:44600/0 (socket says 172.21.15.107:44600) 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f6307fff640 1 -- 172.21.15.107:0/1033370989 learned_addr learned my addr 172.21.15.107:0/1033370989 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 1 -- 172.21.15.107:0/1033370989 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6308072630 con 0x7f6308143c50 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 1 -- 172.21.15.107:0/1033370989 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6308071410 con 0x7f630806fe90 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 1 -- 172.21.15.107:0/1033370989 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080afaa0 con 0x7f63080708a0 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.130+0000 7f630e564640 10 monclient(hunting): authenticate will time out at 1403.398798s 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153717656 0 0) 0x7f63080afaa0 con 0x7f63080708a0 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:52.110 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:52.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:07:52.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f4003520 con 0x7f63080708a0 2024-03-29T11:07:52.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093659171 0 0) 0x7f6308072630 con 0x7f6308143c50 2024-03-29T11:07:52.111 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63080afaa0 con 0x7f6308143c50 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.131+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266480686 0 0) 0x7f6308071410 con 0x7f630806fe90 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6308072630 con 0x7f630806fe90 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332875218 0 0) 0x7f62f4003520 con 0x7f63080708a0 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 >> v1:172.21.15.129:6789/0 conn(0x7f630806fe90 legacy=0x7f6308143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 >> v1:172.21.15.107:6789/0 conn(0x7f6308143c50 legacy=0x7f630814b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:52.112 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 1 monclient: found mon.noname-b 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080ac8b0 con 0x7f63080708a0 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: _finish_auth 0 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6300001a10 con 0x7f63080708a0 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: dump: 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:52.113 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f630e564640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f630e564640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6300001b70 con 0x7f63080708a0 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 1 -- 172.21.15.107:0/1033370989 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f6300001fc0 con 0x7f63080708a0 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f6306ffd640 10 monclient: dump: 2024-03-29T11:07:52.114 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f630e564640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:52.115 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.132+0000 7f630e564640 4 set_mon_vals no callback set 2024-03-29T11:07:52.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.133+0000 7f630e564640 10 monclient: shutdown 2024-03-29T11:07:52.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.133+0000 7f630e564640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:52.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.133+0000 7f630e564640 1 -- 172.21.15.107:0/1033370989 >> v1:172.21.15.107:6790/0 conn(0x7f63080708a0 legacy=0x7f63080af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:52.121 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.133+0000 7f630e564640 1 -- 172.21.15.107:0/1033370989 shutdown_connections 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.134+0000 7f630e564640 1 -- 172.21.15.107:0/1033370989 >> 172.21.15.107:0/1033370989 conn(0x7f63080675b0 msgr2=0x7f63080b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.134+0000 7f630e564640 1 -- 172.21.15.107:0/1033370989 shutdown_connections 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.134+0000 7f630e564640 1 -- 172.21.15.107:0/1033370989 wait complete. 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.135+0000 7f630e564640 10 monclient: build_initial_monmap 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.135+0000 7f630e564640 10 monclient: monmap: 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:07:52.122 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.136+0000 7f630e564640 1 librados: starting msgr at 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.136+0000 7f630e564640 1 librados: starting objecter 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.136+0000 7f630e564640 1 Processor -- start 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.136+0000 7f630e564640 1 -- start start 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.136+0000 7f630e564640 1 librados: setting wanted keys 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.136+0000 7f630e564640 1 librados: calling monclient init 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.136+0000 7f630e564640 10 monclient: init 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.137+0000 7f630e564640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.137+0000 7f630e564640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f630e564640 10 monclient(hunting): picked mon.noname-a con 0x7f630806fe90 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f630e564640 10 monclient(hunting): picked mon.noname-b con 0x7f63080708a0 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f630e564640 10 monclient(hunting): picked mon.noname-c con 0x7f6308143c50 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:52.123 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f6307fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f630806fe90 0x7f6308140030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55048/0 (socket says 172.21.15.107:55048) 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f6307fff640 1 -- 172.21.15.107:0/3246717449 learned_addr learned my addr 172.21.15.107:0/3246717449 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6308142f80 con 0x7f630806fe90 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6308143110 con 0x7f63080708a0 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630813dde0 con 0x7f6308143c50 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f630e564640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f630e564640 10 monclient(hunting): authenticate will time out at 1403.406269s 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f6304ff9640 10 client.?.objecter ms_handle_connect 0x7f630806fe90 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f6304ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f6304ff9640 10 client.?.objecter ms_handle_connect 0x7f6308143c50 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.138+0000 7f6304ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376437058 0 0) 0x7f630813dde0 con 0x7f6308143c50 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:07:52.124 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e8004eb0 con 0x7f6308143c50 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304450492 0 0) 0x7f6308142f80 con 0x7f630806fe90 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f630813dde0 con 0x7f630806fe90 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 client.?.objecter ms_handle_connect 0x7f63080708a0 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2045212811 0 0) 0x7f62e8004eb0 con 0x7f6308143c50 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6308142f80 con 0x7f6308143c50 2024-03-29T11:07:52.125 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f62f80016b0 con 0x7f6308143c50 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): dump: 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:52.126 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1752568600 0 0) 0x7f6308142f80 con 0x7f6308143c50 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.139+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 >> v1:172.21.15.129:6789/0 conn(0x7f63080708a0 legacy=0x7f6308140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 >> v1:172.21.15.107:6789/0 conn(0x7f630806fe90 legacy=0x7f6308140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 1 monclient: found mon.c 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630813df70 con 0x7f6308143c50 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 10 monclient: _finish_auth 0 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f630e564640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:07:52.127 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f630e564640 10 monclient: _renew_subs 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f630e564640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6308143320 con 0x7f6308143c50 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f630e564640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f630e564640 10 monclient: _renew_subs 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f630e564640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63080b5410 con 0x7f6308143c50 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f630e564640 1 librados: init done 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62f80035f0 con 0x7f6308143c50 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f62f8003900 con 0x7f6308143c50 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f6304ff9640 10 monclient: dump: 2024-03-29T11:07:52.128 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.140+0000 7f63067fc640 4 set_mon_vals no callback set 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.141+0000 7f630e564640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.141+0000 7f630e564640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.142+0000 7f630e564640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:52.129 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.142+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec005080 con 0x7f6308143c50 2024-03-29T11:07:52.130 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.143+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f62f80198d0 con 0x7f6308143c50 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.143+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2565443581 0 0) 0x7f62f80837f0 con 0x7f6308143c50 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.143+0000 7f6304ff9640 10 client.4673.objecter ms_dispatch 0x7f6308000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.143+0000 7f6304ff9640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.143+0000 7f6304ff9640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.143+0000 7f6304ff9640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.143+0000 7f6307fff640 10 monclient: get_auth_request con 0x7f62e806dd10 auth_method 0 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.144+0000 7f6304ff9640 10 client.4673.objecter ms_handle_connect 0x7f62e806dd10 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.152+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62f804f0d0 con 0x7f6308143c50 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.152+0000 7f6304ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:52.131 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.152+0000 7f6304ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:52.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.521+0000 7f630e564640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:52.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.521+0000 7f630e564640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:52.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.521+0000 7f630e564640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:52.500 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.521+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62ec005330 con 0x7f6308143c50 2024-03-29T11:07:52.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.522+0000 7f6304ff9640 1 -- 172.21.15.107:0/3246717449 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3225893291) 0x7f62f8051000 con 0x7f6308143c50 2024-03-29T11:07:52.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.522+0000 7f6304ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:52.501 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.522+0000 7f6304ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:52.502 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:07:52.502 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T11:07:15.551396+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:07:15.551396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:52.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 10 librados: watch_flush enter 2024-03-29T11:07:52.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 10 librados: watch_flush exit 2024-03-29T11:07:52.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T11:07:52.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 10 client.4673.objecter successfully canceled tick 2024-03-29T11:07:52.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 >> v1:172.21.15.107:6816/882194755 conn(0x7f62e806dd10 legacy=0x7f62e80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:52.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 10 monclient: shutdown 2024-03-29T11:07:52.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:52.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 >> v1:172.21.15.107:6790/0 conn(0x7f6308143c50 legacy=0x7f63081dda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:52.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 shutdown_connections 2024-03-29T11:07:52.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 >> 172.21.15.107:0/3246717449 conn(0x7f63080675b0 msgr2=0x7f63080b0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:52.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.526+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 shutdown_connections 2024-03-29T11:07:52.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.527+0000 7f630e564640 1 -- 172.21.15.107:0/3246717449 wait complete. 2024-03-29T11:07:52.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:52.527+0000 7f630e564640 1 librados: shutdown 2024-03-29T11:07:52.519 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:59.326 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:59.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.498+0000 7f85f3006640 10 monclient: get_monmap_and_config 2024-03-29T11:07:59.477 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.498+0000 7f85f3006640 10 monclient: build_initial_monmap 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.498+0000 7f85f3006640 10 monclient: monmap: 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:07:59.499850+0000 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:07:59.499850+0000 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:07:59.478 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:59.479 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.500+0000 7f85f3006640 1 Processor -- start 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.500+0000 7f85f3006640 1 -- start start 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.500+0000 7f85f3006640 10 monclient: init 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 10 monclient(hunting): picked mon.noname-b con 0x7f85ec070170 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 10 monclient(hunting): picked mon.noname-a con 0x7f85ec0713f0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 10 monclient(hunting): picked mon.noname-c con 0x7f85ec070550 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec13dbd0 con 0x7f85ec0713f0 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec13ff70 con 0x7f85ec070550 2024-03-29T11:07:59.480 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec140250 con 0x7f85ec070170 2024-03-29T11:07:59.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:59.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.501+0000 7f85f3006640 10 monclient(hunting): authenticate will time out at 1410.769546s 2024-03-29T11:07:59.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.503+0000 7f85f0da2640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f85ec070170 0x7f85ec13d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35638/0 (socket says 172.21.15.107:35638) 2024-03-29T11:07:59.481 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.503+0000 7f85f0da2640 1 -- 172.21.15.107:0/562270336 learned_addr learned my addr 172.21.15.107:0/562270336 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:59.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.505+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630791883 0 0) 0x7f85ec140250 con 0x7f85ec070170 2024-03-29T11:07:59.486 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85dc003520 con 0x7f85ec070170 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898261806 0 0) 0x7f85ec13ff70 con 0x7f85ec070550 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ec140250 con 0x7f85ec070550 2024-03-29T11:07:59.487 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626635119 0 0) 0x7f85d8001040 con 0x7f85ec0713f0 2024-03-29T11:07:59.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:59.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:59.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:59.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:07:59.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ec13ff70 con 0x7f85ec0713f0 2024-03-29T11:07:59.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 <== mon.1 v1:172.21.15.129:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2182189182 0 0) 0x7f85ec140250 con 0x7f85ec070550 2024-03-29T11:07:59.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 >> v1:172.21.15.107:6790/0 conn(0x7f85ec070170 legacy=0x7f85ec13d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:59.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 >> v1:172.21.15.107:6789/0 conn(0x7f85ec0713f0 legacy=0x7f85ec1406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:59.488 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 monclient: found mon.noname-c 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.129:6789/0 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 --> v1:172.21.15.129:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec140df0 con 0x7f85ec070550 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.506+0000 7f85eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85f3006640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85f3006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 <== mon.1 v1:172.21.15.129:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f85e4001970 con 0x7f85ec070550 2024-03-29T11:07:59.489 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:59.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85eb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:59.490 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85eb7fe640 10 monclient: dump: 2024-03-29T11:07:59.490 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:59.490 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:59.490 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.490 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.490 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:59.491 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:59.491 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:59.491 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:59.491 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:59.491 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:59.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85f3006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:59.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 <== mon.1 v1:172.21.15.129:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85e4001b90 con 0x7f85ec070550 2024-03-29T11:07:59.491 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:59.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85f3006640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:59.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.507+0000 7f85f3006640 4 set_mon_vals no callback set 2024-03-29T11:07:59.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85eb7fe640 1 -- 172.21.15.107:0/562270336 <== mon.1 v1:172.21.15.129:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f85e4001f20 con 0x7f85ec070550 2024-03-29T11:07:59.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:59.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:59.492 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85eb7fe640 10 monclient: dump: 2024-03-29T11:07:59.492 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:59.493 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:59.493 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.493 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.493 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:59.493 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:59.493 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:59.494 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:59.494 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:59.494 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:59.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85f3006640 10 monclient: shutdown 2024-03-29T11:07:59.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85f3006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:59.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85f3006640 1 -- 172.21.15.107:0/562270336 >> v1:172.21.15.129:6789/0 conn(0x7f85ec070550 legacy=0x7f85ec14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:59.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85f3006640 1 -- 172.21.15.107:0/562270336 shutdown_connections 2024-03-29T11:07:59.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85f3006640 1 -- 172.21.15.107:0/562270336 >> 172.21.15.107:0/562270336 conn(0x7f85ec067610 msgr2=0x7f85ec0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:59.494 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85f3006640 1 -- 172.21.15.107:0/562270336 shutdown_connections 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.508+0000 7f85f3006640 1 -- 172.21.15.107:0/562270336 wait complete. 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.509+0000 7f85f3006640 10 monclient: build_initial_monmap 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.509+0000 7f85f3006640 10 monclient: monmap: 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:07:59.504 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.509+0000 7f85f3006640 1 librados: starting msgr at 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.509+0000 7f85f3006640 1 librados: starting objecter 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.509+0000 7f85f3006640 1 Processor -- start 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.509+0000 7f85f3006640 1 -- start start 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.509+0000 7f85f3006640 1 librados: setting wanted keys 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.509+0000 7f85f3006640 1 librados: calling monclient init 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.509+0000 7f85f3006640 10 monclient: init 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 10 monclient(hunting): picked mon.noname-c con 0x7f85ec070170 addr v1:172.21.15.107:6790/0 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 10 monclient(hunting): picked mon.noname-b con 0x7f85ec070550 addr v1:172.21.15.129:6789/0 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 10 monclient(hunting): picked mon.noname-a con 0x7f85ec0713f0 addr v1:172.21.15.107:6789/0 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec139ef0 con 0x7f85ec0713f0 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec13ab70 con 0x7f85ec070550 2024-03-29T11:07:59.505 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec138b70 con 0x7f85ec070170 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.510+0000 7f85f3006640 10 monclient(hunting): authenticate will time out at 1410.778925s 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.513+0000 7f85f15a3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f85ec0713f0 0x7f85ec1ee810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56167/0 (socket says 172.21.15.107:56167) 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.514+0000 7f85f0da2640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7f85ec070170 0x7f85ec139690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:35980/0 (socket says 172.21.15.107:35980) 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.514+0000 7f85f0da2640 1 -- 172.21.15.107:0/176804588 learned_addr learned my addr 172.21.15.107:0/176804588 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 client.?.objecter ms_handle_connect 0x7f85ec070170 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 client.?.objecter ms_handle_connect 0x7f85ec0713f0 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 client.?.objecter ms_handle_connect 0x7f85ec070550 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111882566 0 0) 0x7f85ec138b70 con 0x7f85ec070170 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:59.506 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d0004eb0 con 0x7f85ec070170 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410875477 0 0) 0x7f85ec139ef0 con 0x7f85ec0713f0 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ec138b70 con 0x7f85ec0713f0 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.1 v1:172.21.15.129:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766428390 0 0) 0x7f85ec13ab70 con 0x7f85ec070550 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:07:59.507 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.129:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ec139ef0 con 0x7f85ec070550 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1583917862 0 0) 0x7f85d0004eb0 con 0x7f85ec070170 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85ec13ab70 con 0x7f85ec070170 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.515+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2405956223 0 0) 0x7f85ec138b70 con 0x7f85ec0713f0 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85d0004eb0 con 0x7f85ec0713f0 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f85e0001a70 con 0x7f85ec070170 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient(hunting): dump: 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:59.508 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f85e40018b0 con 0x7f85ec0713f0 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient(hunting): dump: 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:59.509 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2740921626 0 0) 0x7f85ec13ab70 con 0x7f85ec070170 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 >> v1:172.21.15.129:6789/0 conn(0x7f85ec070550 legacy=0x7f85ec13a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 >> v1:172.21.15.107:6789/0 conn(0x7f85ec0713f0 legacy=0x7f85ec1ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 1 monclient: found mon.c 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec138d00 con 0x7f85ec070170 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient: _finish_auth 0 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.516+0000 7f85e97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:07:59.510 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85f3006640 5 monclient: authenticate success, global_id 4679 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85f3006640 10 monclient: _renew_subs 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85f3006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.107:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec1e72d0 con 0x7f85ec070170 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85f3006640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85f3006640 10 monclient: _renew_subs 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85f3006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.107:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec0b50d0 con 0x7f85ec070170 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.517+0000 7f85f3006640 1 librados: init done 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.519+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85e0001cf0 con 0x7f85ec070170 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.519+0000 7f85e97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.519+0000 7f85ea7fc640 4 set_mon_vals no callback set 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.520+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7f85e0004c30 con 0x7f85ec070170 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.520+0000 7f85e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:59.511 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.520+0000 7f85e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.520+0000 7f85e97fa640 10 monclient: dump: 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.520+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7f85e001ac00 con 0x7f85ec070170 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.520+0000 7f85f3006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.520+0000 7f85f3006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:59.512 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.520+0000 7f85f3006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.520+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85cc005080 con 0x7f85ec070170 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.521+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2565443581 0 0) 0x7f85e0084b70 con 0x7f85ec070170 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.521+0000 7f85ebfff640 10 monclient: get_auth_request con 0x7f85d006de80 auth_method 0 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.521+0000 7f85e97fa640 10 client.4679.objecter ms_dispatch 0x7f85ec000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.521+0000 7f85e97fa640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.521+0000 7f85e97fa640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.521+0000 7f85e97fa640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.522+0000 7f85e97fa640 10 client.4679.objecter ms_handle_connect 0x7f85d006de80 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.529+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85e00504d0 con 0x7f85ec070170 2024-03-29T11:07:59.513 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.529+0000 7f85e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:59.514 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.529+0000 7f85e97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:59.833 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.854+0000 7f85f3006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:59.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.854+0000 7f85f3006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:59.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.854+0000 7f85f3006640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.107:6790/0 2024-03-29T11:07:59.834 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.854+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 --> v1:172.21.15.107:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85cc005330 con 0x7f85ec070170 2024-03-29T11:07:59.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.856+0000 7f85e97fa640 1 -- 172.21.15.107:0/176804588 <== mon.2 v1:172.21.15.107:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3225893291) 0x7f85e0052400 con 0x7f85ec070170 2024-03-29T11:07:59.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.856+0000 7f85e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:59.835 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.856+0000 7f85e97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:59.835 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:07:59.835 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T11:07:15.551396+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:07:15.551396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:59.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.859+0000 7f85f3006640 10 librados: watch_flush enter 2024-03-29T11:07:59.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.859+0000 7f85f3006640 10 librados: watch_flush exit 2024-03-29T11:07:59.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.859+0000 7f85f3006640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-29T11:07:59.838 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.859+0000 7f85f3006640 10 client.4679.objecter successfully canceled tick 2024-03-29T11:07:59.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.859+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 >> v1:172.21.15.107:6816/882194755 conn(0x7f85d006de80 legacy=0x7f85d0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:59.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.860+0000 7f85f3006640 10 monclient: shutdown 2024-03-29T11:07:59.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.860+0000 7f85f3006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:59.839 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.860+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 >> v1:172.21.15.107:6790/0 conn(0x7f85ec070170 legacy=0x7f85ec139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:59.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.860+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 shutdown_connections 2024-03-29T11:07:59.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.860+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 >> 172.21.15.107:0/176804588 conn(0x7f85ec067610 msgr2=0x7f85ec13d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:59.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.860+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 shutdown_connections 2024-03-29T11:07:59.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.860+0000 7f85f3006640 1 -- 172.21.15.107:0/176804588 wait complete. 2024-03-29T11:07:59.840 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:07:59.861+0000 7f85f3006640 1 librados: shutdown 2024-03-29T11:07:59.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:06.660 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:06.792 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.813+0000 7fbc17e4d640 10 monclient: get_monmap_and_config 2024-03-29T11:08:06.792 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.813+0000 7fbc17e4d640 10 monclient: build_initial_monmap 2024-03-29T11:08:06.792 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.813+0000 7fbc17e4d640 10 monclient: monmap: 2024-03-29T11:08:06.792 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:08:06.792 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:08:06.792 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T11:08:06.814530+0000 2024-03-29T11:08:06.793 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T11:08:06.814530+0000 2024-03-29T11:08:06.793 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:06.793 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:08:06.793 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:08:06.793 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-c 2024-03-29T11:08:06.793 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-b 2024-03-29T11:08:06.793 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:08:06.793 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.814+0000 7fbc17e4d640 1 Processor -- start 2024-03-29T11:08:06.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.815+0000 7fbc17e4d640 1 -- start start 2024-03-29T11:08:06.794 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.815+0000 7fbc17e4d640 10 monclient: init 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 10 monclient(hunting): picked mon.noname-a con 0x7fbc100b0770 addr v1:172.21.15.107:6789/0 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 10 monclient(hunting): picked mon.noname-c con 0x7fbc1014a700 addr v1:172.21.15.129:6789/0 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 10 monclient(hunting): picked mon.noname-b con 0x7fbc100ac5a0 addr v1:172.21.15.107:6790/0 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc100aba70 con 0x7fbc100b0770 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc100aecb0 con 0x7fbc1014a700 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc100af150 con 0x7fbc100ac5a0 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc17e4d640 10 monclient(hunting): authenticate will time out at 1418.084224s 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc163ea640 1 --1- >> v1:172.21.15.107:6790/0 conn(0x7fbc100ac5a0 0x7fbc10151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6790/0 says I am v1:172.21.15.107:42272/0 (socket says 172.21.15.107:42272) 2024-03-29T11:08:06.795 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.816+0000 7fbc163ea640 1 -- 172.21.15.107:0/2683677122 learned_addr learned my addr 172.21.15.107:0/2683677122 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:08:06.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.817+0000 7fbc14be7640 1 -- 172.21.15.107:0/2683677122 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226840365 0 0) 0x7fbc100af150 con 0x7fbc100ac5a0 2024-03-29T11:08:06.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.817+0000 7fbc14be7640 10 monclient(hunting): _negotiate 2024-03-29T11:08:06.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.817+0000 7fbc14be7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:06.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.817+0000 7fbc14be7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:06.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.817+0000 7fbc14be7640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:08:06.796 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.817+0000 7fbc14be7640 1 -- 172.21.15.107:0/2683677122 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbfc003520 con 0x7fbc100ac5a0 2024-03-29T11:08:06.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.818+0000 7fbc14be7640 1 -- 172.21.15.107:0/2683677122 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1000758975 0 0) 0x7fbbfc003520 con 0x7fbc100ac5a0 2024-03-29T11:08:06.797 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.818+0000 7fbc14be7640 1 -- 172.21.15.107:0/2683677122 >> v1:172.21.15.129:6789/0 conn(0x7fbc1014a700 legacy=0x7fbc100abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:08:06.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 1 -- 172.21.15.107:0/2683677122 >> v1:172.21.15.107:6789/0 conn(0x7fbc100b0770 legacy=0x7fbc10149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:06.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 10 monclient: _finish_hunting 0 2024-03-29T11:08:06.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 1 monclient: found mon.noname-b 2024-03-29T11:08:06.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:06.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.107:6790/0 2024-03-29T11:08:06.798 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 1 -- 172.21.15.107:0/2683677122 --> v1:172.21.15.107:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc100ac9d0 con 0x7fbc100ac5a0 2024-03-29T11:08:06.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 10 monclient: _finish_auth 0 2024-03-29T11:08:06.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 10 monclient: _check_auth_tickets 2024-03-29T11:08:06.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:06.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc17e4d640 5 monclient: authenticate success, global_id 4682 2024-03-29T11:08:06.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc17e4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:06.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 1 -- 172.21.15.107:0/2683677122 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbc08001a10 con 0x7fbc100ac5a0 2024-03-29T11:08:06.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:06.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:06.799 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.819+0000 7fbc14be7640 10 monclient: dump: 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.820+0000 7fbc17e4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.820+0000 7fbc14be7640 1 -- 172.21.15.107:0/2683677122 <== mon.2 v1:172.21.15.107:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc08001ca0 con 0x7fbc100ac5a0 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.820+0000 7fbc14be7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.820+0000 7fbc17e4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:06.800 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.820+0000 7fbc17e4d640 4 set_mon_vals no callback set 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.820+0000 7fbc14be7640 1 -- 172.21.15.107:0/2683677122 <== mon.2 v1:172.21.15.107:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbc08002030 con 0x7fbc100ac5a0 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.820+0000 7fbc14be7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.820+0000 7fbc14be7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.820+0000 7fbc14be7640 10 monclient: dump: 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:08:06.801 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:08:06.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.821+0000 7fbc17e4d640 10 monclient: shutdown 2024-03-29T11:08:06.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.821+0000 7fbc17e4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:06.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.821+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2683677122 >> v1:172.21.15.107:6790/0 conn(0x7fbc100ac5a0 legacy=0x7fbc10151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:06.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.821+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2683677122 shutdown_connections 2024-03-29T11:08:06.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.821+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2683677122 >> 172.21.15.107:0/2683677122 conn(0x7fbc1006b660 msgr2=0x7fbc1006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:06.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.821+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2683677122 shutdown_connections 2024-03-29T11:08:06.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.821+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2683677122 wait complete. 2024-03-29T11:08:06.811 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.822+0000 7fbc17e4d640 10 monclient: build_initial_monmap 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.822+0000 7fbc17e4d640 10 monclient: monmap: 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:epoch 0 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:last_changed 0.000000 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:created 0.000000 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.noname-a 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.noname-b 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.noname-c 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.822+0000 7fbc17e4d640 1 librados: starting msgr at 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.822+0000 7fbc17e4d640 1 librados: starting objecter 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.822+0000 7fbc17e4d640 1 Processor -- start 2024-03-29T11:08:06.812 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 1 -- start start 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 1 librados: setting wanted keys 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 1 librados: calling monclient init 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 10 monclient: init 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 10 monclient(hunting): picked mon.noname-b con 0x7fbc100ac5a0 addr v1:172.21.15.129:6789/0 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 10 monclient(hunting): picked mon.noname-c con 0x7fbc100b0770 addr v1:172.21.15.107:6790/0 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 10 monclient(hunting): picked mon.noname-a con 0x7fbc1014a700 addr v1:172.21.15.107:6789/0 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc101280d0 con 0x7fbc1014a700 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 1 -- --> v1:172.21.15.129:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc10127d30 con 0x7fbc100ac5a0 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 1 -- --> v1:172.21.15.107:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc10147b40 con 0x7fbc100b0770 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc17e4d640 10 monclient(hunting): authenticate will time out at 1418.091569s 2024-03-29T11:08:06.813 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc163ea640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbc1014a700 0x7fbc10128d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47732/0 (socket says 172.21.15.107:47732) 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.823+0000 7fbc163ea640 1 -- 172.21.15.107:0/2415040689 learned_addr learned my addr 172.21.15.107:0/2415040689 (peer_addr_for_me v1:172.21.15.107:0/0) 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 client.?.objecter ms_handle_connect 0x7fbc100b0770 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 client.?.objecter ms_handle_connect 0x7fbc1014a700 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052659421 0 0) 0x7fbc101280d0 con 0x7fbc1014a700 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:06.814 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf4004eb0 con 0x7fbc1014a700 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.2 v1:172.21.15.107:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888894082 0 0) 0x7fbc10147b40 con 0x7fbc100b0770 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.824+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 --> v1:172.21.15.107:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc101280d0 con 0x7fbc100b0770 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2816326246 0 0) 0x7fbbf4004eb0 con 0x7fbc1014a700 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 --> v1:172.21.15.107:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc10147b40 con 0x7fbc1014a700 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbc080018d0 con 0x7fbc1014a700 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 10 monclient(hunting): dump: 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:08:06.815 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.2 v1:172.21.15.107:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034165720 0 0) 0x7fbc101280d0 con 0x7fbc100b0770 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 --> v1:172.21.15.107:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbf4004eb0 con 0x7fbc100b0770 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.2 v1:172.21.15.107:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbc00001790 con 0x7fbc100b0770 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:06.816 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 10 monclient(hunting): dump: 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 941370512 0 0) 0x7fbc10147b40 con 0x7fbc1014a700 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 >> v1:172.21.15.107:6790/0 conn(0x7fbc100b0770 legacy=0x7fbc101285f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.825+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 >> v1:172.21.15.129:6789/0 conn(0x7fbc100ac5a0 legacy=0x7fbc10146e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:08:06.817 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 1 monclient: found mon.a 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc10147cd0 con 0x7fbc1014a700 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 10 monclient: _finish_auth 0 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc17e4d640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc17e4d640 10 monclient: _renew_subs 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc17e4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 --> v1:172.21.15.107:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc101df5e0 con 0x7fbc1014a700 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc17e4d640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc17e4d640 10 monclient: _renew_subs 2024-03-29T11:08:06.818 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc17e4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 --> v1:172.21.15.107:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc101deda0 con 0x7fbc1014a700 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc17e4d640 1 librados: init done 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc08001a30 con 0x7fbc1014a700 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.827+0000 7fbc07fff640 4 set_mon_vals no callback set 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.828+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2701852341 0 0) 0x7fbc08005110 con 0x7fbc1014a700 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.828+0000 7fbc067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.828+0000 7fbc067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.828+0000 7fbc067fc640 10 monclient: dump: 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:epoch 1 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:fsid cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:last_changed 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:created 2024-03-29T10:56:32.488595+0000 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:06.819 INFO:teuthology.misc.health.smithi107.stderr:election_strategy: 1 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:0: v1:172.21.15.107:6789/0 mon.a 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:1: v1:172.21.15.129:6789/0 mon.b 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2: v1:172.21.15.107:6790/0 mon.c 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr: 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.830+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1422326265 0 0) 0x7fbc0801b0e0 con 0x7fbc1014a700 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.830+0000 7fbc17e4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.830+0000 7fbc17e4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.830+0000 7fbc17e4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.830+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf8005080 con 0x7fbc1014a700 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.831+0000 7fbc15be9640 10 monclient: get_auth_request con 0x7fbbf406de40 auth_method 0 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.831+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2565443581 0 0) 0x7fbc08084d60 con 0x7fbc1014a700 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.831+0000 7fbc067fc640 10 client.4590.objecter ms_dispatch 0x7fbc10000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.831+0000 7fbc067fc640 3 client.4590.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.831+0000 7fbc067fc640 3 client.4590.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:08:06.820 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.832+0000 7fbc067fc640 20 client.4590.objecter dump_active .. 0 homeless 2024-03-29T11:08:06.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.832+0000 7fbc067fc640 10 client.4590.objecter ms_handle_connect 0x7fbbf406de40 2024-03-29T11:08:06.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.839+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc080506a0 con 0x7fbc1014a700 2024-03-29T11:08:06.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.839+0000 7fbc067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:06.821 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:06.839+0000 7fbc067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:07.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.148+0000 7fbc17e4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:07.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.148+0000 7fbc17e4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:07.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.148+0000 7fbc17e4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.107:6789/0 2024-03-29T11:08:07.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.148+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 --> v1:172.21.15.107:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbf8005330 con 0x7fbc1014a700 2024-03-29T11:08:07.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.149+0000 7fbc067fc640 1 -- 172.21.15.107:0/2415040689 <== mon.0 v1:172.21.15.107:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3225893291) 0x7fbc08084410 con 0x7fbc1014a700 2024-03-29T11:08:07.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.149+0000 7fbc067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:07.128 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.149+0000 7fbc067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:07.129 INFO:teuthology.misc.health.smithi107.stdout: 2024-03-29T11:08:07.129 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"cbfc8126-6e9a-4c5e-9f59-8b71fd32f0ab","created":"2024-03-29T10:57:10.622334+0000","modified":"2024-03-29T11:07:15.551396+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:07:15.551396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9629f157-44b7-489b-9c0a-2f18a8223dc2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f67bee6c-2441-47fe-87e6-7067a828fad9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"e75d2a72-7727-4449-bcc3-d14b92fa0ce8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a40725b6-723a-44b6-adde-604d836b6805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7c47c-8a6e-49ce-a532-0c6e8a353a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"641286c5-474a-40d0-9b3f-a8fec463271f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"22d61d7c-904b-4a1e-9555-7929b313179e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d8e2a8ab-8a6e-4781-b029-8c6277133458","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.152+0000 7fbc17e4d640 10 librados: watch_flush enter 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.152+0000 7fbc17e4d640 10 librados: watch_flush exit 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.152+0000 7fbc17e4d640 20 client.4590.objecter shutdown clearing up homeless session... 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.152+0000 7fbc17e4d640 10 client.4590.objecter successfully canceled tick 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.153+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 >> v1:172.21.15.107:6816/882194755 conn(0x7fbbf406de40 legacy=0x7fbbf4070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.153+0000 7fbc17e4d640 10 monclient: shutdown 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.153+0000 7fbc17e4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.153+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 >> v1:172.21.15.107:6789/0 conn(0x7fbc1014a700 legacy=0x7fbc10128d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.153+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 shutdown_connections 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.153+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 >> 172.21.15.107:0/2415040689 conn(0x7fbc1006b660 msgr2=0x7fbc1006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.153+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 shutdown_connections 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.153+0000 7fbc17e4d640 1 -- 172.21.15.107:0/2415040689 wait complete. 2024-03-29T11:08:07.132 INFO:teuthology.misc.health.smithi107.stderr:2024-03-29T11:08:07.153+0000 7fbc17e4d640 1 librados: shutdown 2024-03-29T11:08:07.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:07.146 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:08:07.148 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:08:07.148 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:08:07.149 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.188 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:08:07.189 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:08:07.189 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.271 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:08:07.271 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:08:07.271 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.304 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:08:07.305 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:08:07.305 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.379 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:08:07.379 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:08:07.379 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.411 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:08:07.412 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:08:07.412 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.491 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:08:07.491 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:08:07.491 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.560 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:08:07.561 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:08:07.561 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.629 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:08:07.630 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:08:07.630 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:08:07.630 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.656 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:08:07.656 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:08:07.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.694 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:08:07.695 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:08:07.695 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:08:07.695 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.708 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:08:07.708 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:08:07.708 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.759 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:08:07.760 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:08:07.760 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:08:07.771 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:08:07.771 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:08:07.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:08:07.833 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T11:08:07.833 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:08:10.526 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T11:08:10.526 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:08:10.619 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T11:08:10.620 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:08:10.678 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T11:08:10.678 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:08:10.760 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T11:08:10.761 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:08:14.376 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T11:08:14.377 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:08:14.480 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T11:08:14.481 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:08:14.553 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T11:08:14.553 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:08:14.612 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:08:14.614 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/data/mon.a.tgz 2024-03-29T11:08:14.614 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-03-29T11:08:14.632 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.SIBWitLXug 2024-03-29T11:08:14.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.SIBWitLXug 2024-03-29T11:08:14.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.SIBWitLXug 2024-03-29T11:08:14.903 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.SIBWitLXug is 211KB 2024-03-29T11:08:14.968 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.SIBWitLXug 2024-03-29T11:08:14.983 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/data/mon.c.tgz 2024-03-29T11:08:14.983 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-03-29T11:08:15.039 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.OhM715RPBz 2024-03-29T11:08:15.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.OhM715RPBz 2024-03-29T11:08:15.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.OhM715RPBz 2024-03-29T11:08:15.223 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.OhM715RPBz is 211KB 2024-03-29T11:08:15.287 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.OhM715RPBz 2024-03-29T11:08:15.301 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/data/mon.b.tgz 2024-03-29T11:08:15.302 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-03-29T11:08:15.320 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.cm5EgcAWZr 2024-03-29T11:08:15.320 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cm5EgcAWZr 2024-03-29T11:08:15.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.cm5EgcAWZr 2024-03-29T11:08:15.525 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.cm5EgcAWZr is 211KB 2024-03-29T11:08:15.591 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.cm5EgcAWZr 2024-03-29T11:08:15.606 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:08:15.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:08:15.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:08:15.688 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:08:15.727 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:08:15.727 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:08:15.729 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/remote/smithi107/crash 2024-03-29T11:08:15.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:08:15.756 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/remote/smithi129/crash 2024-03-29T11:08:15.757 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:08:15.793 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:08:15.793 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:08:15.801 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:08:15.824 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38533.log 2024-03-29T11:08:15.825 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40579.log 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41520.log 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40579.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38533.log.gz 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39838.log 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40579.log.gz 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41520.log.gz 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40423.log 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39799.log 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39838.log.gz 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40423.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2024-03-29T11:08:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40306.log 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41793.log 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39799.log: /var/log/ceph/ceph-client.admin.40306.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40423.log.gz 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41208.log 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41793.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39799.log.gz 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41052.log 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40306.log.gz 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38847.log 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41208.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41793.log.gz 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41208.log.gz 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40501.log 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41052.log.gz 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38769.log 2024-03-29T11:08:15.854 INFO:teuthology.orchestra.run.smithi107.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38847.log.gz 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40501.log: gzip -5 --verbose 85.8% -- -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz /var/log/ceph/ceph-client.admin.39721.log 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41676.log 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40501.log.gz 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41910.log 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38769.log.gz 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40895.log 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39721.log.gz 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41676.log.gz 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40189.log 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40540.log 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41910.log.gz 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-03-29T11:08:15.855 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40895.log.gz 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40189.log.gz 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40736.log 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40540.log.gz 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41247.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39643.log 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40736.log: gzip 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz -5 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39042.log 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41169.log 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40736.log.gz 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39042.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.39643.log.gz 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39354.log 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41325.log 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41169.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39042.log.gz 2024-03-29T11:08:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40657.log 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41169.log.gz 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39472.log 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40657.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41325.log.gz 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40618.log 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40657.log.gz 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40345.log 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39472.log.gz 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5/var/log/ceph/ceph-client.admin.40618.log: --verbose -- /var/log/ceph/ceph-client.admin.38494.log 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40345.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40856.log 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40618.log.gz 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40345.log.gz 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40267.log 2024-03-29T11:08:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38494.log: /var/log/ceph/ceph-client.admin.40856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41754.log 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35168.log 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40856.log.gz 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41754.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40267.log.gz 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40072.log 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41754.log.gz 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39589.log 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38964.log 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35168.log.gz 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39589.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40072.log.gz 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38611.log 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38964.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39589.log.gz 2024-03-29T11:08:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38611.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38964.log.gz 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39760.log 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38611.log.gz 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.1.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41559.log 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39511.log 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39955.log 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39760.log.gz 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39511.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41559.log.gz 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39237.log 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40462.log 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39955.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39511.log.gz 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39159.log 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr: 86.1%/var/log/ceph/ceph-client.admin.40462.log: -- replaced with /var/log/ceph/ceph-client.admin.39955.log.gz 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39994.log 2024-03-29T11:08:15.859 INFO:teuthology.orchestra.run.smithi107.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39237.log.gz 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41832.log 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40462.log.gz 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39003.log 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39994.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39159.log.gz 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41832.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39994.log.gz 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39003.log: gzip -5 --verbose -- 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41832.log.gz 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr: /var/log/ceph/ceph-client.admin.39550.log 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40775.log 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39003.log.gz 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39315.log 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39550.log.gz 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35087.log 2024-03-29T11:08:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39315.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40775.log.gz 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39081.log 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39315.log.gz 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39081.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35087.log.gz 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40111.log 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39081.log.gz 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-mgr.y.log: 86.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39120.log 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38808.log 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40111.log: /var/log/ceph/ceph-client.admin.39120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41598.log 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40111.log.gz 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38808.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39120.log.gz 2024-03-29T11:08:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38808.log.gz 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40974.log 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41598.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40697.log: /var/log/ceph/ceph-client.admin.34772.log 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41286.log 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41598.log.gz 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34772.log: 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40974.log.gz 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.34772.log.gz 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38572.log 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41286.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40150.log 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38572.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41286.log.gz/var/log/ceph/ceph-client.admin.40150.log: 2024-03-29T11:08:15.862 INFO:teuthology.orchestra.run.smithi107.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38572.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:08:15.863 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T11:08:15.863 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40150.log.gz 2024-03-29T11:08:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:08:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39877.log 2024-03-29T11:08:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38455.log 2024-03-29T11:08:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.39877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40228.log 2024-03-29T11:08:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38455.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39877.log.gz 2024-03-29T11:08:15.863 INFO:teuthology.orchestra.run.smithi107.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38455.log.gz 2024-03-29T11:08:15.865 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:08:15.865 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34608.log 2024-03-29T11:08:15.865 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:08:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41715.log 2024-03-29T11:08:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40033.log 2024-03-29T11:08:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41715.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40228.log.gz 2024-03-29T11:08:15.867 INFO:teuthology.orchestra.run.smithi107.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41715.log.gz 2024-03-29T11:08:15.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34648.log 2024-03-29T11:08:15.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:08:15.872 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34608.log.gz 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.34648.log: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34688.log 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34648.log.gz 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34568.log 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.34688.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.34568.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34688.log.gz 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.34528.log 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mgr.x.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34568.log.gz 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34728.log 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.34528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34448.log 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.34728.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34528.log.gz 2024-03-29T11:08:15.873 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34728.log.gz 2024-03-29T11:08:15.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.34448.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:08:15.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34488.log 2024-03-29T11:08:15.874 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34448.log.gz 2024-03-29T11:08:15.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-client.admin.34488.log: 2024-03-29T11:08:15.874 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34488.log.gz 2024-03-29T11:08:15.876 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2024-03-29T11:08:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38730.log 2024-03-29T11:08:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40384.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40033.log.gz 2024-03-29T11:08:15.879 INFO:teuthology.orchestra.run.smithi107.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2024-03-29T11:08:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38886.log 2024-03-29T11:08:15.879 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-03-29T11:08:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38886.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38730.log.gz 2024-03-29T11:08:15.880 INFO:teuthology.orchestra.run.smithi107.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38886.log.gz 2024-03-29T11:08:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28940.log 2024-03-29T11:08:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-03-29T11:08:15.880 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.tmp-client.admin.28940.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28940.log.gz 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gz 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39916.log 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41130.log 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39916.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39916.log.gz 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40814.log 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38650.log 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.40814.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41130.log.gz 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40814.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41871.log 2024-03-29T11:08:15.887 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41637.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38650.log.gz 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38689.log 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39432.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41871.log.gz 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39682.log 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39198.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38689.log.gz 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41364.log 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.39682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38925.log 2024-03-29T11:08:15.888 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.41364.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39682.log.gz 2024-03-29T11:08:15.889 INFO:teuthology.orchestra.run.smithi107.stderr: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:08:15.890 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41364.log.gz 2024-03-29T11:08:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.38925.log: /var/log/ceph/ceph-mon.c.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38925.log.gz 2024-03-29T11:08:15.914 INFO:teuthology.orchestra.run.smithi107.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:08:27.525 INFO:teuthology.orchestra.run.smithi107.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:08:27.967 INFO:teuthology.orchestra.run.smithi129.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:08:28.038 INFO:teuthology.orchestra.run.smithi129.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:08:28.359 INFO:teuthology.orchestra.run.smithi129.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:08:31.241 INFO:teuthology.orchestra.run.smithi107.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:08:31.889 INFO:teuthology.orchestra.run.smithi129.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:08:31.993 INFO:teuthology.orchestra.run.smithi107.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:08:32.068 INFO:teuthology.orchestra.run.smithi107.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:09:03.453 INFO:teuthology.orchestra.run.smithi107.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:09:16.045 INFO:teuthology.orchestra.run.smithi107.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:09:16.047 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-29T11:09:16.047 INFO:teuthology.orchestra.run.smithi107.stderr:real 1m0.235s 2024-03-29T11:09:16.047 INFO:teuthology.orchestra.run.smithi107.stderr:user 2m41.304s 2024-03-29T11:09:16.047 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m5.797s 2024-03-29T11:09:24.269 INFO:teuthology.orchestra.run.smithi129.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:09:24.271 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-29T11:09:24.271 INFO:teuthology.orchestra.run.smithi129.stderr:real 1m8.419s 2024-03-29T11:09:24.271 INFO:teuthology.orchestra.run.smithi129.stderr:user 1m56.650s 2024-03-29T11:09:24.271 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m4.349s 2024-03-29T11:09:24.272 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:09:24.273 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/remote/smithi107/log 2024-03-29T11:09:24.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:09:48.029 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/remote/smithi129/log 2024-03-29T11:09:48.046 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:10:06.750 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:10:06.958 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=db3204eee45b401b9078d5eb37cbb638 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:10:06.962 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:10:06.970 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:10:07.010 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:10:07.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:10:07.030 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:10:07.101 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:10:07.108 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:10:07.109 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:10:07.112 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:10:07.125 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:10:07.126 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:10:07.129 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:10:07.130 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-29T11:10:07.130 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +416us[ +416us] +/- 71ms 2024-03-29T11:10:07.130 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -181us[ -188us] +/- 66ms 2024-03-29T11:10:07.130 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 -186us[ -186us] +/- 66ms 2024-03-29T11:10:07.130 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:10:07.132 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:10:07.132 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-29T11:10:07.132 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 +402us[ +402us] +/- 71ms 2024-03-29T11:10:07.132 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -172us[ -172us] +/- 66ms 2024-03-29T11:10:07.133 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -189us[ -189us] +/- 66ms 2024-03-29T11:10:07.133 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:10:07.133 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:10:07.148 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:10:07.149 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:10:07.177 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:10:07.246 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:10:07.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:10:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:07.342 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:10:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:07.371 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:10:07.380 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:10:07.484 INFO:teuthology.task.internal:Duration was 1211.318173 seconds 2024-03-29T11:10:07.484 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:10:07.519 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:10:07.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:10:07.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:10:07.564 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:07.565 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:07.853 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:10:07.854 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-03-29T11:10:07.854 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:10:07.874 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-03-29T11:10:07.874 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:10:07.914 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:10:07.914 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:10:07.917 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:10:07.982 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:10:07.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:10:07.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:10:08.118 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:10:08.128 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:10:08.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:10:08.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:10:08.182 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:10:08.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:10:08.197 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:10:08.220 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-03-29T11:10:08.249 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-03-29T11:10:08.281 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:10:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:08.301 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:10:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:08.335 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:10:08.344 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:10:08.346 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/remote/smithi107 2024-03-29T11:10:08.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:10:08.382 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667/remote/smithi129 2024-03-29T11:10:08.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:10:08.417 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:10:08.417 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:08.420 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:08.467 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:10:08.477 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:10:08.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:10:08.495 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:10:08.495 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:10:08.498 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:10:08.509 INFO:teuthology.orchestra.run.smithi107.stdout: 266033 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:10 /home/ubuntu/cephtest 2024-03-29T11:10:08.521 INFO:teuthology.orchestra.run.smithi129.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:10 /home/ubuntu/cephtest 2024-03-29T11:10:08.522 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:10:08.533 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:10:08.595 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:10:08.619 DEBUG:teuthology.nuke:shortname: smithi107 2024-03-29T11:10:08.619 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:10:08.642 DEBUG:teuthology.nuke:shortname: smithi129 2024-03-29T11:10:08.642 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:10:08.646 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:41:44.155932', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDFzZxd2kvuz30bPI7xAlhJL5tJ7HLRwgTx6Tm6UbCbL2tMstDFr0/oYOlXARa7OJEEd80SMqSR3NCkqnlii78Q='} 2024-03-29T11:10:08.669 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622667', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:41:44.157686', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHNHbCoWq3xgvmIDZlynuq9vF0SpeCEjLpTRbMDfDt5Q/x/9Wy9L9q5yC27U+4CL6J7Pr+MJ8ognYTedVfMR8So='} 2024-03-29T11:10:08.695 INFO:teuthology.orchestra.console.smithi107:Power off 2024-03-29T11:10:08.696 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:10:08.719 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-03-29T11:10:08.720 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:10:08.739 INFO:teuthology.orchestra.console.smithi129:Power off 2024-03-29T11:10:08.739 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:10:08.749 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-29T11:10:08.763 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2024-03-29T11:10:08.763 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:10:08.791 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-29T11:10:12.750 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:10:12.792 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:10:12.861 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-29T11:10:12.903 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-29T11:10:16.866 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:10:16.904 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:10:16.977 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-29T11:10:17.014 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2024-03-29T11:10:17.115 INFO:teuthology.orchestra.console.smithi129:Power off completed 2024-03-29T11:10:20.982 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:10:21.093 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-03-29T11:10:21.193 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-03-29T11:10:21.298 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1211.3181729316711 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=db3204eee45b401b9078d5eb37cbb638 status: fail success: false 2024-03-29T11:10:21.298 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:10:21.380 INFO:teuthology.run:FAIL